Hi arrow-dev, I have a few questions/suggestions for updates to the Layout.md document. I'd like feedback and if people agree I can make the necessary updates.
1. For completeness it might be useful to add a statement that the byte order (endianness) is platform native. Rationale: The first use-cases for arrow are targeted for IPC so specifying a byte order that potentially isn't native adds unnecessary overhead. 2. The document specifies null bitmaps should have a length padded to a multiple of 8 bytes to avoid word alignment concerns. I assume the concern is having a subsequent value buffer that isn't properly aligned. Are there other concerns that make the padding worthwhile? If not, it might pay to define the term "contiguous memory buffer" (already mentioned in the document) as a "contiguous memory region" that starts at an address that is a multiple of 8 and not require the padding (both values and bitmaps are stored in their own contiguous memory buffers). This would ensure that odd-length primitive byte Arrays won't mess up alignment either. 3. Unions contain type arrays defined as: "An array of unsigned integers, enumerated from 0 corresponding to each type, with the smallest byte width capable of representing the number of types in the union." Making these integers signed eases compatibility with Java (something we are already doing for other uses of integers). According to Jacques, Arrow's Java implementation already works this way. Additionally, in the rare case that it takes more the 2 bytes to represent the number of types in the union, we might want to use the next power of 2-byte width (i.e. 4-bytes). This will make decoding the types easier at the cost of additional memory. 4. The current maximum number of element in a column is 2^31-1. https://issues.apache.org/jira/browse/SPARK-10399 lists a (contrived use-case) of storing images. This could make https://issues.apache.org/jira/browse/ARROW-39 (fixed size row-batches) awkward, since determining the correct number of rows for each batch to avoid overflow would be difficult. There are a few options I can think of to solve this: a. Do nothing (just accept it isn't supported) b. Change length of Arrays to be a 64 bit signed int and offsets for List types to be 64 bit ints. c. Define a new type "LargeList" that has offsets of 64 bit signed integers, an additional list of corresponding lengths (signed 32 bit integers) for each offset and a vector of nested value arrays (all arrays except for the last one are equal in size and their size is a power of 2 to facilitate efficient random access). Each array will only contain complete lists (offset and offset+length, will always reference addresses in the same buffer). This implies there will be some fragmentation/unused space. I think this is unavoidable. Lists spanning different arrays would require a memcpy/memory allocation or it would require returning multiple offsets/lengths and make the client reassemble the List. In the short term, option "a" seems reasonable. In the longer term I prefer option "c". Option "b" would require allocating very large chunks of contiguous memory and it doesn't seem like it would be compatible with the Java (ByteBuffer.allocateDirect takes an int, I am not sure about the Unsafe APIs). Thoughts? Thanks, Micah
