[Format] Is it legal to have a struct array with a shorter length than its children?

2023-05-05 Thread Weston Pace
We allow arrays to have a shorter length than their buffers. Is it also legal for a struct array to have a shorter length than its child arrays? For example, in C++, I can create this today by slicing a struct array: ``` std::shared_ptr my_array = std::dynamic_pointer_cast(array); ASSERT_EQ(m

Re: [Format] Is it legal to have a struct array with a shorter length than its children?

2023-05-05 Thread Gang Wu
IMHO, this is valid. As you have demonstrated in the example, a sliced struct array will result in a length shorter than its child arrays. This kind of flexibility can make it easy to reuse child arrays within the struct array. > Struct: a nested layout consisting of a collection of named child fi

Re: [Format] Is it legal to have a struct array with a shorter length than its children?

2023-05-06 Thread Antoine Pitrou
I agree with Gang. The fact that a struct field may be backed by a physically larger C++ ArrayData is irrelevant, as long as it's logically interpreted as having "the same length". (however, this implementation detail should ideally not leak into IPC or C Data exports) Regards Antoine.

Re: [Format] Is it legal to have a struct array with a shorter length than its children?

2023-05-09 Thread Micah Kornfield
> > I think the `the same length` means the length of the struct array, this is > similar in the case of RecordBatch where the `num_rows` of a RecordBatch > can be different to the length of its fields. I didn't think this is true and I thought we had prior discussions on the mailing list sugges