Hi Rok, Just posted a comment to the doc but wanted to add it here, plus add a little bit of extra info.
Option C looks like by far the strongest option to me. Here's my take: Option A: not backwards compatible, poor encodings -- little upside. Option B: clearly the path we'd take if we were designing a new format, but for parquet as-is it would require a tremendous amount of work, and that alone makes it untenable unless the gain it gives is way better than other options. Option C: Fully backwards compatible, keeps full encodings, small amount of work to implement on readers and writers. The only downside is that we still have to store the rep levels on disk (and load them), but due to the fixed length arrays they compress to almost nothing under the RLE encodings, so the cost is tiny. The extra info I wanted to add is that even for files without anything added to them, you can in fact cheaply detect whether the array in question is of a fixed size by doing some logic on the compressed rle data. On an example dataset I benchmarked (100k rows of 4k-float array features) I measured the decompression time to be 2.3x faster than our baseline reading path. That verification does of course cost some time, and if we had the hint from C in the data we could skip it, giving us another 1.5x. On Wed, Jul 1, 2026 at 11:24 AM Jiayi Wang <[email protected]> wrote: > > ---------- Forwarded message --------- > 发件人： Rok Mihevc <[email protected]> > Date: 2026年6月29日周一 20:13 > Subject: Re: [DISCUSSION] Introduce FIXED_SIZE_LIST logical type > To: <[email protected]> > Cc: <[email protected]> > > > Hi all, > > Based on benchmarks [1] comparing implementation variants, feedback > received via > the design doc [2] and in-person feedback I would like to move forward with > the approach that introduces a new repetition type (Option B in the design > doc and benchmarks) unless there are objections. Either here or on this > week's community call would be a good venue to raise early objections > or provide feedback on how to proceed. > > To show what a new repetition type would look like I've opened a > parquet-format PR [3] and a draft Go implementation [4]. > > [1] https://gist.github.com/rok/fe4785d4a74d2e080cbad73e88cc1bef > [2] > > https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0 > [3] https://github.com/apache/parquet-format/pull/592 > [4] https://github.com/apache/arrow-go/pull/854 > > Rok > > On Sun, Jun 14, 2026 at 10:10 PM Rok Mihevc <[email protected]> wrote: > > > Hi all, > > > > A short update on the progress of this work. State of discussion can be > > seen here [1]. > > I've created a set of naive C++ implementations of the discussed designs; > > see here: https://gist.github.com/rok/fe4785d4a74d2e080cbad73e88cc1bef > > Results should be taken with a grain of salt and more of a directional > > rather than quantitative information. > > > > Personally I'm leaning towards option B because it is quite expressive > > while still providing significant storage and writing performance > > improvement. > > > > [1] > > > https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?usp=sharing > > [2] https://gist.github.com/rok/fe4785d4a74d2e080cbad73e88cc1bef - > > benchmarks > > [3] https://github.com/rok/arrow/pull/53 - option A > > [4] https://github.com/rok/arrow/pull/51 - option B > > [5] https://github.com/rok/arrow/pull/52 - option C > > > > Rok > > > > On Tue, May 5, 2026 at 3:21 PM Rok Mihevc <[email protected]> wrote: > > > >> Hi all, > >> > >> Picking this thread back up. I've put together a design doc outlining > >> three options we've discussed: > >> > >> > https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?usp=sharing > >> > >> * Option A: logical type annotating FIXED_LEN_BYTE_ARRAY. > >> * Option B: new VECTOR repetition type. > >> * Option C: logical type annotating a normal LIST, where a recognizing > >> reader skips rep-level decode and an unknown reader still sees a working > >> LIST. A future revision would let writers omit rep-levels entirely. > >> > >> The document evaluates these against the same requirements and compares > >> them along six axes (backwards compatibility, composability, encoding > >> flexibility, implementation complexity, on-disk overhead, read > >> performance). The doc aims to centralize the discussion and help us > pick a > >> direction. > >> Comments are open. Most useful pushback would be on the requirements > >> (especially the "no-fallback breaks adoption" one). > >> > >> Best, > >> Rok > >> > >> On Tue, Mar 3, 2026 at 8:58 PM Antoine Pitrou <[email protected]> > wrote: > >> > >>> > >>> Hello, > >>> > >>> The downside with this approach is that the top-level "unit" type is > not > >>> the element type. > >>> > >>> For example, if you have a FIXED_SIZE_LIST(FLOAT32, 3), then the > >>> top-level unit type is FIXED_LEN_BYTE_ARRAY(12). This means that > >>> specialized encodings such as BYTE_STREAM_SPLIT, DELTA_BINARY_PACKED or > >>> ALP may either be less efficient (for BYTE_STREAM_SPLIT) or not be > >>> applicable at all (for the latter two). > >>> > >>> I wonder if we can find an approach that doesn't emit repetition levels > >>> but still allows using efficient encodings for the element type. > >>> > >>> Regards > >>> > >>> Antoine. > >>> > >>> > >>> Le 03/03/2026 à 01:13, Rok Mihevc a écrit : > >>> > Hi all, > >>> > > >>> > I'd like to resurrect this thread in light of recent vectors in > Parquet > >>> > discussion [1]. > >>> > There is a (now updated) proposal PR from when the thread was started > >>> that > >>> > has a nice discussion [2]. > >>> > > >>> > TLDR of the current proposal: > >>> > - FIXED_SIZE_LIST annotates a FIXED_LEN_BYTE_ARRAY primitive leaf > with > >>> > FixedSizeListType { type, num_values }. > >>> > - type must be fixed-width and non-array (INT32, INT64, FLOAT, > DOUBLE, > >>> > FIXED_LEN_BYTE_ARRAY); num_values > 0. > >>> > - type_length must match num_values encoded with PLAIN representation > >>> of > >>> > type. > >>> > - If the field is optional, the whole list value may be null; > elements > >>> are > >>> > always non-null. > >>> > - Intentionally not a `LIST` encoding (no def/rep levels). > >>> > - Outer page/column encoding behavior is unchanged (any encoding > valid > >>> for > >>> > `FIXED_LEN_BYTE_ARRAY` remains valid). > >>> > > >>> > I also added explicit validity requirements: writers must not emit > >>> > violating metadata, and readers must treat violating metadata as > >>> invalid. > >>> > > >>> > > >>> > Rok > >>> > > >>> > [1] https://lists.apache.org/thread/nmq7odlbg1p6yx0hg00clzjbc3tb1tc3 > >>> > [2] https://github.com/apache/parquet-format/pull/241 > >>> > > >>> > On Thu, May 16, 2024 at 4:34 AM Jan Finis <[email protected]> wrote: > >>> > > >>> >> I would love to see this! > >>> >> > >>> >> It is an important optimization for vectors, which become more and > >>> more > >>> >> important and ubiquitous for grounding of LLMs. > >>> >> > >>> >> Note however that the logical type route has one drawback: A logical > >>> type > >>> >> may not change the physical representation of values! Thus, if we > make > >>> >> FIXED_SIZE_LIST just a logical type, we would still need to write > >>> R-Levels, > >>> >> as even clients not knowing this logical type need to be able to > >>> decode the > >>> >> column. We could avoid reading the R-Levels and just assume that > each > >>> list > >>> >> has the fixed size, so the read path would be optimized but the > write > >>> path > >>> >> wouldn't. > >>> >> > >>> >> If we want to avoid writing R-Levels altogether, a logical type > >>> doesn't cut > >>> >> it. It needs to be something different. E.g., in the schema, we > could > >>> store > >>> >> an optional `count` for repeated fields. Whenever this count is > >>> present, we > >>> >> would not write R-Levels for this field (or more precisely, this > field > >>> >> would not take part in the R-Level computation, as if it wasn't a > >>> repeated > >>> >> field). This of course is a more intrusive change, as legacy clients > >>> >> couldn't read such columns anymore. > >>> >> > >>> >> I don't know which of the two alternatives is better. I agree with > >>> Gang > >>> >> that we should probably discuss this in a PR. > >>> >> > >>> >> Cheers, > >>> >> Jan > >>> >> > >>> >> > >>> >> Am Mi., 15. Mai 2024 um 14:03 Uhr schrieb Gang Wu <[email protected] > >: > >>> >> > >>> >>> Hi Rok, > >>> >>> > >>> >>> Happy to see you here :) > >>> >>> > >>> >>> According to my past experience, it would be more helpful to open > >>> >>> a PR against the parquet-format repository and post it here. > >>> >>> > >>> >>> Best, > >>> >>> Gang > >>> >>> > >>> >>> On Wed, May 15, 2024 at 7:25 PM Rok Mihevc <[email protected]> > >>> wrote: > >>> >>> > >>> >>>> Hi all, > >>> >>>> > >>> >>>> Arrow recently introduced FixedShapeTensor and VariableShapeTensor > >>> >>>> canonical extension types [1] that use FixedSizeList and > >>> >>> StructArray(List, > >>> >>>> FixedSizeList) as storage respectfully. These are targeted at > >>> machine > >>> >>>> learning and scientific applications that deal with large datasets > >>> and > >>> >>>> would benefit from using Parquet as on disk storage. > >>> >>>> > >>> >>>> However currently FixedSizeList is stored as List in Parquet which > >>> adds > >>> >>>> significant conversion overhead when reading and writing [2]. It > >>> would > >>> >>>> therefore be beneficial to introduce a FIXED_SIZE_LIST logical > type. > >>> >>>> > >>> >>>> I would like to open a discussion on potentially adding > >>> FIXED_SIZE_LIST > >>> >>>> type and prepare a proposal if discussion supports it. > >>> >>>> > >>> >>>> > >>> >>>> Best, > >>> >>>> Rok > >>> >>>> > >>> >>>> [1] > >>> >>>> > >>> >>> > >>> >> > >>> > https://arrow.apache.org/docs/format/CanonicalExtensions.html#official-list > >>> >>>> [2] https://github.com/apache/arrow/issues/34510 > >>> >>>> > >>> >>> > >>> >> > >>> > > >>> > >>> > >>> >
