Re: Question about nested columnar validity

2023-06-29 Thread Weston Pace
>> 2. For StringView and ArrayView, if the parent has `validity = false`. >> If they have `validity = true`, there offset might point to a invalid >> position >I have no idea, but I hope not. Ben Kietzman might want to answer more >precisely here. I think, for view arrays, the offsets

Re: Question about nested columnar validity

2023-06-29 Thread Antoine Pitrou
Le 29/06/2023 à 15:16, wish maple a écrit : Sorry for being misleading. "valid" offset means that: 1. For Binary Like [1] format, and List formats [2], even if the parent has `validity = false`. Their offset should be well-defined. Yes. 2. For StringView and ArrayView, if the parent

Re: Question about nested columnar validity

2023-06-29 Thread Antoine Pitrou
Le 29/06/2023 à 13:42, wish maple a écrit : Thanks all! So, in general: 1. For our Binary Like [1] format, and List formats [2], if the parent is not valid, the offset should still be valid What do you call a "valid" offset?

RE: Question about nested columnar validity

2023-06-29 Thread wish maple
Thanks all! So, in general: 1. For our Binary Like [1] format, and List formats [2], if the parent is not valid, the offset should still be valid 2. For the StringView ListView [3] types arrow is currently working on, if the parent is not valid, the child might has valid content Am I

Re: Question about nested columnar validity

2023-06-29 Thread Felipe Oliveira Carvalho
Values in the `offsets` Buffer of a ListArray can’t be left undefined because the length of a valid entry before a NULL entry is the offset associated with that NULL entry minus the previous offset. The ListViewArray format I’m working on doesn’t have that restriction because all the information

Re: Question about nested columnar validity

2023-06-29 Thread Antoine Pitrou
Le 29/06/2023 à 06:07, Weston Pace a écrit : When a binary array or a list array element is null the cleanest thing to do is to set the offsets to be the same. So, for example, given a list array with 5 elements, if second item is null, the offsets could be 0, 8, 8, 12, 20, 50. Question

Re: Question about nested columnar validity

2023-06-28 Thread Weston Pace
I agree with Antoine but I get easily confused by "valid, as in structurally correct" and "valid, as in not null" so I want to make sure I understand: > The child of a nested > array should be valid itself, independently of the parent's validity bitmap. A child must be "structurally correct"

Re: Question about nested columnar validity

2023-06-28 Thread Antoine Pitrou
Hi! Le 28/06/2023 à 17:03, wish maple a écrit : Hi, By looking at the arrow standard, when it comes to nested structure, like StructArray[1] or FixedListArray[2], when parent is not valid, the correspond child leaves "undefined". If it's a BinaryArray, when when it parent is not valid,