Yes, ListView is an implementation of Velox's ArrayVector [1] ("vector of
arrays"). In Arrow we would naturally refer to them as "array of lists",
but `ListArray` is taken by the existing offset-only list formats.
Following the pattern adopted by other types in Arrow that use offsets and
sizes, we adopt the suffix -View to differentiate list-views from lists.Velox doesn't offer the 64-bit variation, but since Arrow has both List and LargeList, it was natural to pair them with ListView and LargeListView. [2] is a link to the point of a talk by Mark Raasveldt where he describes the DuckDB list representation. Early in the talk, one of the slides [3] mentions how these formats were "co-designed together with Velox team". -- Felipe [1] https://facebookincubator.github.io/velox/develop/vectors.html#arrayvector [2] https://youtu.be/bZOvAKGkzpQ?si=wgSwew3Ck8utteOI&t=1569 [3] https://15721.courses.cs.cmu.edu/spring2023/slides/22-duckdb.pdf On Fri, Sep 29, 2023 at 9:32 AM Raphael Taylor-Davies <r.taylordav...@googlemail.com.invalid> wrote: > Hi Felipe, > > Can I confirm that DuckDB and Velox use the same encoding for these > types, and so we aren't going to run into similar issues as [1]? > > Kind Regards, > > Raphael Taylor-Davies > > [1]: https://lists.apache.org/thread/l8t1vj5x1wdf75mdw3wfjvnxrfy5xomy > > On 29/09/2023 13:09, Felipe Oliveira Carvalho wrote: > > Hello, > > > > I'd like to propose adding ListView and LargeListView arrays to the Arrow > > format. > > Previous discussion in [1][2], columnar format description and > flatbuffers > > changes in [3]. > > > > There are implementations available in both C++ [4] and Go [5]. I'm > working > > on the integration tests which I will push to one of the PR branches > before > > they are merged. I've made a graph illustrating how this addition > affects, > > in a backwards compatible way, the type predicates and inheritance chain > on > > the C++ implementation. [6] > > > > The vote will be open for at least 72 hours not counting the weekend. > > > > [ ] +1 add the proposed ListView and LargeListView types to the Apache > > Arrow format > > [ ] -1 do not add the proposed ListView and LargeListView types to the > > Apache Arrow format > > because... > > > > Sincerely, > > Felipe > > > > [1] https://lists.apache.org/thread/r28rw5n39jwtvn08oljl09d4q2c1ysvb > > [2] https://lists.apache.org/thread/dcwdzhz15fftoyj6xp89ool9vdk3rh19 > > [3] https://github.com/apache/arrow/pull/37877 > > [4] https://github.com/apache/arrow/pull/35345 > > [5] https://github.com/apache/arrow/pull/37468 > > [6] https://gist.github.com/felipecrv/3c02f3784221d946dec1b031c6d400db > > >
