>
> Slicing is part of the C data interface (with the offset member).
OK, so refreshing myself for the C data interface, IIUC, I think one needs
to hack RLE at a parent Array with two children arrays, because otherwise
in general, I don't think I see a way of actually communicating buffer size
at all (unless we want to revisit the API, which might not be worth it).
But it would still require special logic for decoding. For the slice
example I think it could be represented as (changing slice to '6" since
"4" make some things non-obvious):
{
length: 2
offset: 6
rle: {
length: 1 // actually physical length
offset: 2
buffer: [3, 5,8]
}
values: {
length: 1
offset: 2
buffer: [5, 6, 7]
}
}
Does this make sense?
Apologies Weston, if this is what you were getting at, but I think it is
slightly different because you talked about 0 offsets in your example.
> > So a slice at 4 would be:
> > Run ends: [5, 8]
> > Values: [6, 7]
How do you interpret that?
> Naively, that means the logical values [6, 6, 6, 6, 6, 7, 7, 7]
> It doesn't seem right...
I think the key is you need to keep the logical offset around as part of
you metadata so you can do the subtract.
On Thu, Sep 15, 2022 at 1:19 AM Antoine Pitrou <anto...@python.org> wrote:
>
> Le 15/09/2022 à 10:14, Micah Kornfield a écrit :
> > I agree slicing can be tricky here. Since slicing is not part of the
> > specification, maybe there should be two separate discussions here. I'll
> > be honest, I forget exactly how slicing works in the C++ implementation,
> > but is
>
> Slicing is part of the C data interface (with the offset member).
>
> >> Say you have the logical values: [5, 5, 5, 6, 6, 7, 7, 7]
> >>
> >> Run ends: [3, 5, 8]
> >> Values: [5, 6, 7]
> >
> > So a slice at 4 would be:
> > Run ends: [5, 8]
> > Values: [6, 7]
>
> How do you interpret that?
> Naively, that means the logical values [6, 6, 6, 6, 6, 7, 7, 7]
> It doesn't seem right...
>
> > For Lookup of elements one could add the logical offset to the index and
> to
> > the binary search as normal.
>
> Right. But that implies we just have a logical offset and no physical
> offset. Which is probably fine to me, but not what Weston proposed AFAIU
> :-)
>
> > I guess this might be harder to implement based on the current slicing
> > implementation? Or I might be missing something obvious?
>
> Well, in any case, slicing will be more involved for RLE than it is for
> other data types or encodings...
>
> Regards
>
> Antoine.
>
