So if the implementation effort difference is marginal, compatibility issues can be handled with versioning, and vector columns are arguably a niche type (limiting the blast radius), why would we consider a less optimal approach? Are there other reasons we haven't discussed yet?
In either case we need to introduce a logical type, bringing us to about the same time frame for changing the spec. Rok On Mon, Jul 6, 2026 at 3:08 PM Alkis Evlogimenos via dev < [email protected]> wrote: > We have implemented Option C and the work is a couple of hundred lines of > code. I estimate the Option B will have about the same complexity. The > benefit of Option C is that we can ship it asap and we have no backward or > forward compatibility to think about. > > On Mon, Jul 6, 2026 at 2:24 PM Antoine Pitrou <[email protected]> wrote: > > > > > AFAICT, option C has the downside that a optional FIXED_SIZE_LIST would > > have a max definition level of 2, not 1. This can have significant > > effects on the performance of decoding definition levels. > > > > Also, option C is only a "small amount of work to implement" if you > > don't care about optimizing away the unnecessary processing of > > repetition levels. If you do, then it's not obvious that it would be > > less work to implement than option B. > > > > Regards > > > > Antoine. > > > > > > Le 01/07/2026 à 20:35, Andrew McCormick via dev a écrit : > > > Hi Rok, > > > > > > Just posted a comment to the doc but wanted to add it here, plus add a > > > little bit of extra info. > > > > > > Option C looks like by far the strongest option to me. Here's my take: > > > > > > Option A: not backwards compatible, poor encodings -- little upside. > > > Option B: clearly the path we'd take if we were designing a new format, > > but > > > for parquet as-is it would require a tremendous amount of work, and > that > > > alone makes it untenable unless the gain it gives is way better than > > other > > > options. > > > Option C: Fully backwards compatible, keeps full encodings, small > amount > > of > > > work to implement on readers and writers. The only downside is that we > > > still have to store the rep levels on disk (and load them), but due to > > the > > > fixed length arrays they compress to almost nothing under the RLE > > > encodings, so the cost is tiny. > > > > > > The extra info I wanted to add is that even for files without anything > > > added to them, you can in fact cheaply detect whether the array in > > question > > > is of a fixed size by doing some logic on the compressed rle data. On > an > > > example dataset I benchmarked (100k rows of 4k-float array features) I > > > measured the decompression time to be 2.3x faster than our baseline > > reading > > > path. That verification does of course cost some time, and if we had > the > > > hint from C in the data we could skip it, giving us another 1.5x. > > > > > > On Wed, Jul 1, 2026 at 11:24 AM Jiayi Wang <[email protected]> wrote: > > > > > > > > > > > >> > > >> ---------- Forwarded message --------- > > >> 发件人： Rok Mihevc <[email protected]> > > >> Date: 2026年6月29日周一 20:13 > > >> Subject: Re: [DISCUSSION] Introduce FIXED_SIZE_LIST logical type > > >> To: <[email protected]> > > >> Cc: <[email protected]> > > >> > > >> > > >> Hi all, > > >> > > >> Based on benchmarks [1] comparing implementation variants, feedback > > >> received via > > >> the design doc [2] and in-person feedback I would like to move forward > > with > > >> the approach that introduces a new repetition type (Option B in the > > design > > >> doc and benchmarks) unless there are objections. Either here or on > this > > >> week's community call would be a good venue to raise early objections > > >> or provide feedback on how to proceed. > > >> > > >> To show what a new repetition type would look like I've opened a > > >> parquet-format PR [3] and a draft Go implementation [4]. > > >> > > >> [1] https://gist.github.com/rok/fe4785d4a74d2e080cbad73e88cc1bef > > >> [2] > > >> > > >> > > > https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?tab=t.0 > > >> [3] https://github.com/apache/parquet-format/pull/592 > > >> [4] https://github.com/apache/arrow-go/pull/854 > > >> > > >> Rok > > >> > > >> On Sun, Jun 14, 2026 at 10:10 PM Rok Mihevc <[email protected]> > > wrote: > > >> > > >>> Hi all, > > >>> > > >>> A short update on the progress of this work. State of discussion can > be > > >>> seen here [1]. > > >>> I've created a set of naive C++ implementations of the discussed > > designs; > > >>> see here: > https://gist.github.com/rok/fe4785d4a74d2e080cbad73e88cc1bef > > >>> Results should be taken with a grain of salt and more of a > directional > > >>> rather than quantitative information. > > >>> > > >>> Personally I'm leaning towards option B because it is quite > expressive > > >>> while still providing significant storage and writing performance > > >>> improvement. > > >>> > > >>> [1] > > >>> > > >> > > > https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?usp=sharing > > >>> [2] https://gist.github.com/rok/fe4785d4a74d2e080cbad73e88cc1bef - > > >>> benchmarks > > >>> [3] https://github.com/rok/arrow/pull/53 - option A > > >>> [4] https://github.com/rok/arrow/pull/51 - option B > > >>> [5] https://github.com/rok/arrow/pull/52 - option C > > >>> > > >>> Rok > > >>> > > >>> On Tue, May 5, 2026 at 3:21 PM Rok Mihevc <[email protected]> > > wrote: > > >>> > > >>>> Hi all, > > >>>> > > >>>> Picking this thread back up. I've put together a design doc > outlining > > >>>> three options we've discussed: > > >>>> > > >>>> > > >> > > > https://docs.google.com/document/d/1nf30OqK_UqxA4YTEZQszmOBEG56m9M5mp9rIYC2SUWc/edit?usp=sharing > > >>>> > > >>>> * Option A: logical type annotating FIXED_LEN_BYTE_ARRAY. > > >>>> * Option B: new VECTOR repetition type. > > >>>> * Option C: logical type annotating a normal LIST, where a > recognizing > > >>>> reader skips rep-level decode and an unknown reader still sees a > > working > > >>>> LIST. A future revision would let writers omit rep-levels entirely. > > >>>> > > >>>> The document evaluates these against the same requirements and > > compares > > >>>> them along six axes (backwards compatibility, composability, > encoding > > >>>> flexibility, implementation complexity, on-disk overhead, read > > >>>> performance). The doc aims to centralize the discussion and help us > > >> pick a > > >>>> direction. > > >>>> Comments are open. Most useful pushback would be on the requirements > > >>>> (especially the "no-fallback breaks adoption" one). > > >>>> > > >>>> Best, > > >>>> Rok > > >>>> > > >>>> On Tue, Mar 3, 2026 at 8:58 PM Antoine Pitrou <[email protected]> > > >> wrote: > > >>>> > > >>>>> > > >>>>> Hello, > > >>>>> > > >>>>> The downside with this approach is that the top-level "unit" type > is > > >> not > > >>>>> the element type. > > >>>>> > > >>>>> For example, if you have a FIXED_SIZE_LIST(FLOAT32, 3), then the > > >>>>> top-level unit type is FIXED_LEN_BYTE_ARRAY(12). This means that > > >>>>> specialized encodings such as BYTE_STREAM_SPLIT, > DELTA_BINARY_PACKED > > or > > >>>>> ALP may either be less efficient (for BYTE_STREAM_SPLIT) or not be > > >>>>> applicable at all (for the latter two). > > >>>>> > > >>>>> I wonder if we can find an approach that doesn't emit repetition > > levels > > >>>>> but still allows using efficient encodings for the element type. > > >>>>> > > >>>>> Regards > > >>>>> > > >>>>> Antoine. > > >>>>> > > >>>>> > > >>>>> Le 03/03/2026 à 01:13, Rok Mihevc a écrit : > > >>>>>> Hi all, > > >>>>>> > > >>>>>> I'd like to resurrect this thread in light of recent vectors in > > >> Parquet > > >>>>>> discussion [1]. > > >>>>>> There is a (now updated) proposal PR from when the thread was > > started > > >>>>> that > > >>>>>> has a nice discussion [2]. > > >>>>>> > > >>>>>> TLDR of the current proposal: > > >>>>>> - FIXED_SIZE_LIST annotates a FIXED_LEN_BYTE_ARRAY primitive leaf > > >> with > > >>>>>> FixedSizeListType { type, num_values }. > > >>>>>> - type must be fixed-width and non-array (INT32, INT64, FLOAT, > > >> DOUBLE, > > >>>>>> FIXED_LEN_BYTE_ARRAY); num_values > 0. > > >>>>>> - type_length must match num_values encoded with PLAIN > > representation > > >>>>> of > > >>>>>> type. > > >>>>>> - If the field is optional, the whole list value may be null; > > >> elements > > >>>>> are > > >>>>>> always non-null. > > >>>>>> - Intentionally not a `LIST` encoding (no def/rep levels). > > >>>>>> - Outer page/column encoding behavior is unchanged (any encoding > > >> valid > > >>>>> for > > >>>>>> `FIXED_LEN_BYTE_ARRAY` remains valid). > > >>>>>> > > >>>>>> I also added explicit validity requirements: writers must not emit > > >>>>>> violating metadata, and readers must treat violating metadata as > > >>>>> invalid. > > >>>>>> > > >>>>>> > > >>>>>> Rok > > >>>>>> > > >>>>>> [1] > > https://lists.apache.org/thread/nmq7odlbg1p6yx0hg00clzjbc3tb1tc3 > > >>>>>> [2] https://github.com/apache/parquet-format/pull/241 > > >>>>>> > > >>>>>> On Thu, May 16, 2024 at 4:34 AM Jan Finis <[email protected]> > > wrote: > > >>>>>> > > >>>>>>> I would love to see this! > > >>>>>>> > > >>>>>>> It is an important optimization for vectors, which become more > and > > >>>>> more > > >>>>>>> important and ubiquitous for grounding of LLMs. > > >>>>>>> > > >>>>>>> Note however that the logical type route has one drawback: A > > logical > > >>>>> type > > >>>>>>> may not change the physical representation of values! Thus, if we > > >> make > > >>>>>>> FIXED_SIZE_LIST just a logical type, we would still need to write > > >>>>> R-Levels, > > >>>>>>> as even clients not knowing this logical type need to be able to > > >>>>> decode the > > >>>>>>> column. We could avoid reading the R-Levels and just assume that > > >> each > > >>>>> list > > >>>>>>> has the fixed size, so the read path would be optimized but the > > >> write > > >>>>> path > > >>>>>>> wouldn't. > > >>>>>>> > > >>>>>>> If we want to avoid writing R-Levels altogether, a logical type > > >>>>> doesn't cut > > >>>>>>> it. It needs to be something different. E.g., in the schema, we > > >> could > > >>>>> store > > >>>>>>> an optional `count` for repeated fields. Whenever this count is > > >>>>> present, we > > >>>>>>> would not write R-Levels for this field (or more precisely, this > > >> field > > >>>>>>> would not take part in the R-Level computation, as if it wasn't a > > >>>>> repeated > > >>>>>>> field). This of course is a more intrusive change, as legacy > > clients > > >>>>>>> couldn't read such columns anymore. > > >>>>>>> > > >>>>>>> I don't know which of the two alternatives is better. I agree > with > > >>>>> Gang > > >>>>>>> that we should probably discuss this in a PR. > > >>>>>>> > > >>>>>>> Cheers, > > >>>>>>> Jan > > >>>>>>> > > >>>>>>> > > >>>>>>> Am Mi., 15. Mai 2024 um 14:03 Uhr schrieb Gang Wu < > > [email protected] > > >>> : > > >>>>>>> > > >>>>>>>> Hi Rok, > > >>>>>>>> > > >>>>>>>> Happy to see you here :) > > >>>>>>>> > > >>>>>>>> According to my past experience, it would be more helpful to > open > > >>>>>>>> a PR against the parquet-format repository and post it here. > > >>>>>>>> > > >>>>>>>> Best, > > >>>>>>>> Gang > > >>>>>>>> > > >>>>>>>> On Wed, May 15, 2024 at 7:25 PM Rok Mihevc < > [email protected]> > > >>>>> wrote: > > >>>>>>>> > > >>>>>>>>> Hi all, > > >>>>>>>>> > > >>>>>>>>> Arrow recently introduced FixedShapeTensor and > > VariableShapeTensor > > >>>>>>>>> canonical extension types [1] that use FixedSizeList and > > >>>>>>>> StructArray(List, > > >>>>>>>>> FixedSizeList) as storage respectfully. These are targeted at > > >>>>> machine > > >>>>>>>>> learning and scientific applications that deal with large > > datasets > > >>>>> and > > >>>>>>>>> would benefit from using Parquet as on disk storage. > > >>>>>>>>> > > >>>>>>>>> However currently FixedSizeList is stored as List in Parquet > > which > > >>>>> adds > > >>>>>>>>> significant conversion overhead when reading and writing [2]. > It > > >>>>> would > > >>>>>>>>> therefore be beneficial to introduce a FIXED_SIZE_LIST logical > > >> type. > > >>>>>>>>> > > >>>>>>>>> I would like to open a discussion on potentially adding > > >>>>> FIXED_SIZE_LIST > > >>>>>>>>> type and prepare a proposal if discussion supports it. > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> Best, > > >>>>>>>>> Rok > > >>>>>>>>> > > >>>>>>>>> [1] > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>> > > >> > > > https://arrow.apache.org/docs/format/CanonicalExtensions.html#official-list > > >>>>>>>>> [2] https://github.com/apache/arrow/issues/34510 > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>>>> > > >>>>> > > >> > > > > > > > > > >
