[RESULT][VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

2023-10-04 Thread Matt Topol
Hey All, We have 5 binding +1 votes, 2 non-binding +1 votes, and no -1 votes, so the vote passes. Thanks everyone for participating in this and thanks again to Felipe for pushing through this and making it happen! --Matt On Mon, Oct 2, 2023 at 11:50 PM Felipe Oliveira Carvalho <

Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

2023-10-02 Thread Felipe Oliveira Carvalho
pola.rs, for instance, uses only the LargeListType to represent their logical List type. [1] — Felipe [1] https://github.com/pola-rs/polars/blob/e3a491e8e5d8d50f3f5d0420a122a74e1e8194a4/crates/polars-core/src/datatypes/dtype.rs#L238C17-L238C17 On Tue, 3 Oct 2023 at 00:22 Micah Kornfield

Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

2023-10-02 Thread Micah Kornfield
Sorry to chime in late. In practice I'm not sure how much LargeList is used? Are we doing this just for symmetry purposes? Is there a known use-case for it? On Mon, Oct 2, 2023 at 11:20 AM Matt Topol wrote: > Should have expanded my messages, i forgot that i already +1'd this > > d'oh! Sorry

Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

2023-10-02 Thread Matt Topol
Should have expanded my messages, i forgot that i already +1'd this d'oh! Sorry for the spam! --Matt On Mon, Oct 2, 2023 at 2:19 PM Matt Topol wrote: > +1 > > On Mon, Oct 2, 2023 at 8:54 AM Raphael Taylor-Davies > wrote: > >> +1 >> >> On 02/10/2023 13:53, Antoine Pitrou wrote: >> > >> >

Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

2023-10-02 Thread Matt Topol
+1 On Mon, Oct 2, 2023 at 8:54 AM Raphael Taylor-Davies wrote: > +1 > > On 02/10/2023 13:53, Antoine Pitrou wrote: > > > > Hello, > > > > +1 and thanks for working on this! > > > > There'll probably be some minor comments to the format PR, but those > > don't deter from accepting these new

Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

2023-10-02 Thread Raphael Taylor-Davies
+1 On 02/10/2023 13:53, Antoine Pitrou wrote: Hello, +1 and thanks for working on this! There'll probably be some minor comments to the format PR, but those don't deter from accepting these new layouts into the standard. Regards Antoine. Le 29/09/2023 à 14:09, Felipe Oliveira Carvalho

Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

2023-10-02 Thread Antoine Pitrou
Hello, +1 and thanks for working on this! There'll probably be some minor comments to the format PR, but those don't deter from accepting these new layouts into the standard. Regards Antoine. Le 29/09/2023 à 14:09, Felipe Oliveira Carvalho a écrit : Hello, I'd like to propose adding

Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

2023-09-29 Thread Matt Topol
+1, thanks Felipe for your perseverance here! On Fri, Sep 29, 2023, 12:55 PM wish maple wrote: > +1 > > LGTM, thanks! > > Ian Cook 于2023年9月30日周六 00:49写道: > > > +1 (non-binding) > > > > Thanks very much Felipe for your persistence and your commitment to > > addressing the numerous questions and

Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

2023-09-29 Thread David Li
+1 Thanks all for pushing this forward! On Fri, Sep 29, 2023, at 12:55, wish maple wrote: > +1 > > LGTM, thanks! > > Ian Cook 于2023年9月30日周六 00:49写道: > >> +1 (non-binding) >> >> Thanks very much Felipe for your persistence and your commitment to >> addressing the numerous questions and comments

Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

2023-09-29 Thread wish maple
+1 LGTM, thanks! Ian Cook 于2023年9月30日周六 00:49写道: > +1 (non-binding) > > Thanks very much Felipe for your persistence and your commitment to > addressing the numerous questions and comments that have been raised > since the beginning of the discussion on this in April. > > On Fri, Sep 29, 2023

Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

2023-09-29 Thread Ian Cook
+1 (non-binding) Thanks very much Felipe for your persistence and your commitment to addressing the numerous questions and comments that have been raised since the beginning of the discussion on this in April. On Fri, Sep 29, 2023 at 12:34 PM Benjamin Kietzman wrote: > > +1 > > On Fri, Sep 29,

Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

2023-09-29 Thread Benjamin Kietzman
+1 On Fri, Sep 29, 2023 at 10:51 AM Felipe Oliveira Carvalho < felipe...@gmail.com> wrote: > Yes, ListView is an implementation of Velox's ArrayVector [1] ("vector of > arrays"). In Arrow we would naturally refer to them as "array of lists", > but `ListArray` is taken by the existing offset-only

Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

2023-09-29 Thread Felipe Oliveira Carvalho
Yes, ListView is an implementation of Velox's ArrayVector [1] ("vector of arrays"). In Arrow we would naturally refer to them as "array of lists", but `ListArray` is taken by the existing offset-only list formats. Following the pattern adopted by other types in Arrow that use offsets and sizes, we

Re: [VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

2023-09-29 Thread Raphael Taylor-Davies
Hi Felipe, Can I confirm that DuckDB and Velox use the same encoding for these types, and so we aren't going to run into similar issues as [1]? Kind Regards, Raphael Taylor-Davies [1]: https://lists.apache.org/thread/l8t1vj5x1wdf75mdw3wfjvnxrfy5xomy On 29/09/2023 13:09, Felipe Oliveira

[VOTE][Format] Add ListView and LargeListView Arrays to Arrow Format

2023-09-29 Thread Felipe Oliveira Carvalho
Hello, I'd like to propose adding ListView and LargeListView arrays to the Arrow format. Previous discussion in [1][2], columnar format description and flatbuffers changes in [3]. There are implementations available in both C++ [4] and Go [5]. I'm working on the integration tests which I will