+1 (non-binding)
Thanks very much Felipe for your persistence and your commitment to
addressing the numerous questions and comments that have been raised
since the beginning of the discussion on this in April.
On Fri, Sep 29, 2023 at 12:34 PM Benjamin Kietzman <bengil...@gmail.com> wrote:
>
> +1
>
> On Fri, Sep 29, 2023 at 10:51 AM Felipe Oliveira Carvalho <
> felipe...@gmail.com> wrote:
>
> > Yes, ListView is an implementation of Velox's ArrayVector [1] ("vector of
> > arrays"). In Arrow we would naturally refer to them as "array of lists",
> > but `ListArray` is taken by the existing offset-only list formats.
> > Following the pattern adopted by other types in Arrow that use offsets and
> > sizes, we adopt the suffix -View to differentiate list-views from lists.
> >
> > Velox doesn't offer the 64-bit variation, but since Arrow has both List and
> > LargeList, it was natural to pair them with ListView and LargeListView.
> >
> > [2] is a link to the point of a talk by Mark Raasveldt where he describes
> > the DuckDB list representation. Early in the talk, one of the slides [3]
> > mentions how these formats were "co-designed together with Velox team".
> >
> > --
> > Felipe
> >
> > [1]
> > https://facebookincubator.github.io/velox/develop/vectors.html#arrayvector
> > [2] https://youtu.be/bZOvAKGkzpQ?si=wgSwew3Ck8utteOI&t=1569
> > [3] https://15721.courses.cs.cmu.edu/spring2023/slides/22-duckdb.pdf
> >
> > On Fri, Sep 29, 2023 at 9:32 AM Raphael Taylor-Davies
> > <r.taylordav...@googlemail.com.invalid> wrote:
> >
> > > Hi Felipe,
> > >
> > > Can I confirm that DuckDB and Velox use the same encoding for these
> > > types, and so we aren't going to run into similar issues as [1]?
> > >
> > > Kind Regards,
> > >
> > > Raphael Taylor-Davies
> > >
> > > [1]: https://lists.apache.org/thread/l8t1vj5x1wdf75mdw3wfjvnxrfy5xomy
> > >
> > > On 29/09/2023 13:09, Felipe Oliveira Carvalho wrote:
> > > > Hello,
> > > >
> > > > I'd like to propose adding ListView and LargeListView arrays to the
> > Arrow
> > > > format.
> > > > Previous discussion in [1][2], columnar format description and
> > > flatbuffers
> > > > changes in [3].
> > > >
> > > > There are implementations available in both C++ [4] and Go [5]. I'm
> > > working
> > > > on the integration tests which I will push to one of the PR branches
> > > before
> > > > they are merged. I've made a graph illustrating how this addition
> > > affects,
> > > > in a backwards compatible way, the type predicates and inheritance
> > chain
> > > on
> > > > the C++ implementation. [6]
> > > >
> > > > The vote will be open for at least 72 hours not counting the weekend.
> > > >
> > > > [ ] +1 add the proposed ListView and LargeListView types to the Apache
> > > > Arrow format
> > > > [ ] -1 do not add the proposed ListView and LargeListView types to the
> > > > Apache Arrow format
> > > > because...
> > > >
> > > > Sincerely,
> > > > Felipe
> > > >
> > > > [1] https://lists.apache.org/thread/r28rw5n39jwtvn08oljl09d4q2c1ysvb
> > > > [2] https://lists.apache.org/thread/dcwdzhz15fftoyj6xp89ool9vdk3rh19
> > > > [3] https://github.com/apache/arrow/pull/37877
> > > > [4] https://github.com/apache/arrow/pull/35345
> > > > [5] https://github.com/apache/arrow/pull/37468
> > > > [6] https://gist.github.com/felipecrv/3c02f3784221d946dec1b031c6d400db
> > > >
> > >
> >
