I suggest we go with VECTOR for the finite vector to keep the name the same
as other engines/type systems. If we want a non-finite one in the future we
can name that something else or we add fixed size list that will serve the
same purpose without a special name.


On Thu, Oct 8, 2026 at 10:20 PM Ryan Blue <[email protected]> wrote:

> A logical type would:
> - Describe data for users and readers to understand what the file contains
> - Signal that repetition and definition levels will reflect a fixed number
> of elements, which implementations can use to optimize decoding
> - Guarantee that there are no null elements (because the type cannot be
> applied to a list with optional elements)
> - [FINITE_VECTOR] Guarantee that no elements are NaN or Infinity values so
> that the vectors have a finite length and are suitable for distance
> calculations
>
> The finite value restriction is a restriction on writers to prevent bad
> data from being written.
>
> Ignoring logical types that are not understood is a rule for readers. A
> reader can handle vectors and pass it on to another layer. For instance, a
> Parquet reader that doesn't support geometry passes values as `binary`.
> That binary value can be passed to a geometry UDF, even if the underlying
> engine and Parquet reader doesn't have support.
>
> A tool that doesn't understand FINITE_VECTOR can similarly pass a list of
> floats along to a UDF. But it is still important to the user that bad
> values are not present in the data. In your example, bad values are vectors
> that cause the centroid calculation to fail, like vectors with infinite
> length.
>
> Ryan
>
> On Thu, Oct 8, 2026 at 12:54 PM Will Edwards via dev <
> [email protected]>
> wrote:
>
> > Could we clarify the use cases for a VECTOR annotation on LIST<FLOAT>?
> > Specifically, what does it make possible that's impossible today, and
> what
> > does it make easier that's currently hard?
> >
> > I'm struggling to find a strong answer. I'd like every engine to offer
> > vector functions on lists of floats without needing a new logical type.
> >
> > To be clear, I'm still enthusiastic about a vector type that describes
> how
> > to unpack and scale composite representations (quantised codes plus
> scales,
> > codebooks, etc.), in the direction ML is going. That adds information
> > engines don't have. A label on LIST<FLOAT> doesn't: engines already
> > understand the physical type, and many already offer vector functions on
> > it. I don't see a performance benefit, and NaN/null/Inf checks are cheap
> > enough that engines should do them anyway. The label also doesn't say
> what
> > you can do with the column. Knowing that a column holds embeddings
> doesn't
> > say how they can be used, just as zipcode and age are both integers but
> > multiplying them is meaningless.
> >
> > My concern is the opposite effect. We're establishing that engines should
> > ignore logical types they don't understand. So a tool that doesn't know
> > VECTOR will read the column as LIST<FLOAT>, let the user compute
> centroids
> > (say), and write plain LIST<FLOAT> back out. If other engines restrict
> > embedding functions to VECTOR-annotated columns, that file stops working
> > with them, through no fault of the user. An annotation that doesn't
> change
> > what the data is shouldn't be able to make it unusable.
> >
> > Am I missing a use case?
> >
> > Best, Will
> >
> > On Thu, 8 Oct 2026 at 19:53, Steven Wu <[email protected]> wrote:
> >
> > > I also think a FINITE_VECTOR type brings a lot of values to many use
> > cases.
> > > It is would be great to make progress on the FINITE_VECTOR, and not to
> > fit
> > > every use cases into the FIXED_LENGTH_LIST.
> > >
> > > On Mon, Oct 5, 2026 at 1:26 PM Ryan Blue <[email protected]> wrote:
> > >
> > > > Going through this thread again, it looks like there are a few things
> > > that
> > > > people generally agree on:
> > > >
> > > > - In the short term, we want to introduce a logical type or types; a
> > > better
> > > > physical representation can be done later. We can focus on logical
> > types.
> > > > - There is precedent in logical types for restricting values, like
> UTF8
> > > > strings to contain only UTF-8 codepoints. I think we also agree that
> we
> > > > could restrict vector elements to non-nested or primitive types.
> > > > - There are many examples of databases that restrict their vector
> types
> > > to
> > > > non-null, finite elements.
> > > > - Some people want to have write-time guarantees that vectors have
> only
> > > > non-null, finite elements
> > > > - Some people want to extend this work to include tensors, shorter
> > > floating
> > > > point representations, variable lengths, NaN or Infinity elements,
> etc.
> > > >
> > > > It sounds like we are nearly there, but have two halves of the
> > community
> > > > that care about different things. That just means we should have
> > multiple
> > > > types. It's entirely reasonable to want a type that allows NaN
> values,
> > > and
> > > > it's also reasonable for those of us building databases, table
> > standards,
> > > > and vector indexes to care about guarantees that elements are finite.
> > > > There's more than enough examples to show that this is a legitimate
> use
> > > > case.
> > > >
> > > > Since I don't see the concerns of the different use cases changing,
> it
> > > > looks like we need to find a compromise to move forward. I propose
> that
> > > we
> > > > introduce a FINITE_VECTOR type and separately a FIXED_LENGTH_LIST or
> > > > another VECTOR type that allows infinite and NaN values.
> > > >
> > > > Can we agree on a compromise? Is there a strong reason to try to fit
> > > these
> > > > use cases into a single type? What if that means not making progress?
> > > >
> > > > Ryan
> > > >
> > > > On Mon, Oct 5, 2026 at 12:15 PM Philipp Fischbeck via dev <
> > > > [email protected]> wrote:
> > > >
> > > > > Thinking about this more, I am also fine with pulling out the
> > “NaN/Inf”
> > > > > requirement. Either into a type boolean (my preference), or putting
> > no
> > > > > restrictions on the Parquet level, allowing all IEEE 754 float
> > numbers.
> > > > If
> > > > > some top-level application wants to put additional constraints on
> the
> > > > data
> > > > > type, like Iceberg not allowing NaN/infinity, it can still do so
> one
> > > > layer
> > > > > above. It seems like this was the biggest point of discussion
> > > remaining.
> > > > > All these options work for me, and we should move forward with this
> > > > > proposal.
> > > > >
> > > > > Concerning additional element types like bf16, they can be
> introduced
> > > as
> > > > > Parquet types (at least in the current LIST-based spec proposal).
> > > > >
> > > > > Concerning Will’s suggestion around representing VECTOR as a
> variable
> > > > > length list with a MIN/MAX_LENGTH metadata restriction: I think
> that
> > a
> > > > > dedicated logical VECTOR type still provides enough benefits by
> > > > describing
> > > > > the semantics of the data, and allowing to capture future changes
> > > around
> > > > > statistics. In that case, the length is better captured as an exact
> > > > length
> > > > > for this type specifically. I’m also not sure about other use cases
> > for
> > > > > MIN_LENGTH, so introducing a new type constraint system just for
> the
> > > > > MIN_LENGTH=MAX_LENGTH case for vectors doesn’t have enough value in
> > my
> > > > > view.
> > > > >
> > > > > Best,
> > > > > Philipp
> > > > >
> > > > >
> > > > > On Thu, Oct 1, 2026 at 9:00 AM Will Edwards via dev <
> > > > > [email protected]>
> > > > > wrote:
> > > > >
> > > > > > +1 Sebastian, a vector type suggests usage beyond normalized
> > > embeddings
> > > > > and
> > > > > > engines really aren't going to be able to elide all the checks
> > > anyway.
> > > > > >
> > > > > > Antoine, you suggested we explore metadata solutions too :)
> > > > > >
> > > > > > Today, a database supporting CHAR(N) or VARCHAR(n) can't express
> > this
> > > > in
> > > > > > Parquet.  Engines can write opaque blobs they understand in the
> > > > metadata
> > > > > > (e.g. Arrow does) so the extra type info round-trips, but this
> > isn't
> > > > > > standardized, portable, nor interoperable.
> > > > > >
> > > > > > So perhaps standardizing constraint metadata or new Thrift fields
> > (my
> > > > > pref)
> > > > > > can generically support VECTOR(N) whilst also being more broadly
> > > useful
> > > > > for
> > > > > > other use cases and data types?
> > > > > >
> > > > > > Imagine metadata or Thrift fields to tag the MIN_LENGTH and
> > > MAX_LENGTH
> > > > > for
> > > > > > Parquet's variable-length physcial types.  These would count
> > logical
> > > > > units
> > > > > > (e.g. UTF8 code points for STRING) rather than bytes.
> > > > > >
> > > > > > Then one engine's VECTOR(N) can be written to Parquet as LIST(N)
> > with
> > > > > > MIN_LENGTH = MAX_LENGTH = N and then readers - even different
> > > engines -
> > > > > can
> > > > > > pick up these constraints if they support doing so.  We don't
> > > actually
> > > > > need
> > > > > > a logical type to express a VECTOR then.
> > > > > >
> > > > > > This also allows standard SQL's CHAR(N) and VARCHAR(N) to be
> > > expressed
> > > > on
> > > > > > UTF8 columns etc.
> > > > > >
> > > > > > This also seems generally useful in the future as we approach
> > > > expressing
> > > > > > other tensor data shapes and types.
> > > > > >
> > > > > > This won't change the mechanics: the storage type doesn't enforce
> > > these
> > > > > > constraints and engines must check them. It's previously
> > established
> > > > that
> > > > > > performance is just a software problem and there is no
> fundamental
> > > > > > performance advantage to the fixed size types in Dremel-like
> > formats
> > > > like
> > > > > > Parquet.
> > > > > >
> > > > > > Best,
> > > > > > Will
> > > > > >
> > > > > > On Wed, 30 Sept 2026 at 16:08, Sebastian Baunsgaard <
> > > > > [email protected]
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > After listening in on the last meeting Wednesday, I have been
> > > > thinking
> > > > > > > about this quite a bit, and thought I would share my 2 cents.
> > > > > > >
> > > > > > > I suggest not requiring VECTOR elements to be finite, as
> proposed
> > > in
> > > > > the
> > > > > > PR
> > > > > > > <https://github.com/apache/parquet-format/pull/624>.
> > > > > > >
> > > > > > > While Divjot and others note the specialization for UTF-8,
> > Parquet
> > > > > > already
> > > > > > > defines DOUBLE and FLOAT as IEEE encodings
> > > > > > > <
> > > > https://github.com/apache/parquet-format/blob/master/README.md#types
> > > > > >.
> > > > > > >
> > > > > > > Making NaN or +/-Inf illegal inside VECTOR would be a new
> > > restriction
> > > > > on
> > > > > > > those same underlying physical types. Scalar FLOAT/DOUBLE
> columns
> > > > have
> > > > > > > never had such a rule.
> > > > > > >
> > > > > > > It would also not match how the systems that actually produce
> > these
> > > > > > vectors
> > > > > > > treat floats. Most of them use IEEE floats, e.g. Spark, DuckDB,
> > > > NumPy,
> > > > > > > PyTorch, TensorFlow, or lower level CUDA, oneMKL, and OpenBLAS.
> > > > > > >
> > > > > > > pgvector's "all elements must be finite" rule is an insert
> > policy.
> > > > > While
> > > > > > I
> > > > > > > think Parquet as a general-purpose format should allow them out
> > of
> > > > the
> > > > > > box.
> > > > > > > It is a user system decision to disallow writing these values
> or
> > to
> > > > > throw
> > > > > > > errors when encountering them.
> > > > > > >
> > > > > > > A type-level ban also does not keep Inf/NaN out of vector math.
> > > > Finite
> > > > > > > coordinates still overflow. For instance: magnitude =
> > ||[FLOAT_MAX,
> > > > > > > FLOAT_MAX]|| where every coordinate is finite overflows to
> +Inf.
> > > > > PyTorch
> > > > > > > documents the same overflow: tensor([1e20, 1e20]).norm() is inf
> > in
> > > > > fp32.
> > > > > > >
> > > > > > > A generated MAGNITUDE(vec) column would then be Inf while the
> > > VECTOR
> > > > > > itself
> > > > > > > was required to be finite. The ban only forbids storing
> specials
> > in
> > > > the
> > > > > > > coordinates, and does not prevent even simple downstream
> > > computations
> > > > > > from
> > > > > > > producing them.
> > > > > > >
> > > > > > > I suggest keeping VECTOR as the type and add an optional bool
> > > > > > > VectorType.finite (default false). Engines that want
> > pgvector-like
> > > > > > > rejection can require the flag = true. Engines that use NumPy /
> > > > > PyTorch /
> > > > > > > Spark arrays can keep representable IEEE values. For the same
> > > > reason, I
> > > > > > do
> > > > > > > not favor adding yet another type called FINITE_VECTOR.
> > > > > > >
> > > > > > > Best,
> > > > > > > Sebastian
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Sep 29, 2026 at 2:03 PM Divjot Arora via dev <
> > > > > > > [email protected]>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I agree with moving forward with the proposal and renaming
> > VECTOR
> > > > to
> > > > > > > > FINITE_VECTOR or something
> > > > > > > > similar if others feel that better communicates its intent. I
> > see
> > > > > value
> > > > > > > in
> > > > > > > > having a vector type that guarantees
> > > > > > > > its values are mathematically-correct vectors for a few
> > reasons:
> > > > > > > >
> > > > > > > > 1. As others have mentioned, this is similar to the STRING
> > > logical
> > > > > type
> > > > > > > > mandating UTF-8.
> > > > > > > > 2. It is not possible to rely on statistics for detecting NaN
> > or
> > > > > +-inf
> > > > > > > > values as all statistics are optional.
> > > > > > > > To my knowledge, there is no precedent for requiring that
> > writers
> > > > > emit
> > > > > > > > stats for specific logical types.
> > > > > > > > 3. Without statistics, NaN/inf values could flow through the
> > > engine
> > > > > and
> > > > > > > > block them for specific functionality.
> > > > > > > > This seems to be Lance's behavior. However, I don't think
> this
> > is
> > > > as
> > > > > > easy
> > > > > > > > as it sounds and Lance has run
> > > > > > > > into several bugs due to this ([1] [2] for examples). Their
> > > Python
> > > > > [3]
> > > > > > > and
> > > > > > > > Rust [4] APIs default to erroring on
> > > > > > > > vectors that contain NaN values.
> > > > > > > >
> > > > > > > > I agree that Parquet is a general-purpose format, but I don't
> > > think
> > > > > > this
> > > > > > > > implies the lack of domain-specific types.
> > > > > > > > Instead, this broadens the use cases where Parquet would be
> > > > > considered
> > > > > > a
> > > > > > > > good fit. I also fully agree with
> > > > > > > > continuing work on a general fixed-size list type if we
> believe
> > > > there
> > > > > > are
> > > > > > > > non-vector use cases that would benefit
> > > > > > > > from such guarantees at the storage layer.
> > > > > > > >
> > > > > > > > [1] https://github.com/lance-format/lance/pull/3648
> > > > > > > > [2] https://github.com/lance-format/lance/issues/8607 (see
> > > point 2
> > > > > in
> > > > > > > > "root
> > > > > > > > cause")
> > > > > > > > [3]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/python/python/lancedb/table.py#L4807-L4911
> > > > > > > > [4]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lancedb/lancedb/blob/0327618a62483feb6dca2336549845e5887cd48f/rust/lancedb/src/table/add_data.rs#L105-L110
> > > > > > > >
> > > > > > > > On Mon, Sep 28, 2026 at 6:19 PM Philipp Fischbeck via dev <
> > > > > > > > [email protected]> wrote:
> > > > > > > >
> > > > > > > > > I also favor moving forward with the proposal as is. I see
> > > strong
> > > > > > value
> > > > > > > > and
> > > > > > > > > use cases for the numeric vector type, and I agree with the
> > > > > > > restrictions
> > > > > > > > > around NaN/Inf values and element types in general.
> Precedent
> > > > > exists
> > > > > > in
> > > > > > > > > other types that restrict which physical type values are
> > > > logically
> > > > > > > valid
> > > > > > > > > (e.g., the geospatial types needing to be WKB-encoded).  A
> > > > general
> > > > > > > > > fixed-size list also has value, but I think it makes sense
> to
> > > > > > separate
> > > > > > > > the
> > > > > > > > > two types. The semantics are relevant, and I can see how
> the
> > > two
> > > > > > types
> > > > > > > > > might diverge further in the future regarding statistics or
> > > > > > encodings.
> > > > > > > > >
> > > > > > > > > I would be okay with renaming the vector type, but it seems
> > > like
> > > > > the
> > > > > > > > VECTOR
> > > > > > > > > name is already used across other engines mentioned in this
> > > > thread.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Philipp
> > > > > > > > >
> > > > > > > > > On Sat, Sep 26, 2026 at 4:37 PM Gang Wu <[email protected]>
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > To separate orthogonal concerns, I feel we should treat
> the
> > > > > general
> > > > > > > > > > FixedSizeList as the underlying physical layout, and
> > separate
> > > > it
> > > > > > from
> > > > > > > > the
> > > > > > > > > > discussion of a specialized VECTOR (or broader ML tensor)
> > > > logical
> > > > > > > type.
> > > > > > > > > >
> > > > > > > > > > Currently, the immediate need is to define a specialized
> > > VECTOR
> > > > > > type
> > > > > > > so
> > > > > > > > > > that different engines and formats have a common semantic
> > > > > contract
> > > > > > > for
> > > > > > > > > > interoperability.
> > > > > > > > > >
> > > > > > > > > > If the main point of contention is just the NaN/Inf/null
> > > > > > limitations,
> > > > > > > > > maybe
> > > > > > > > > > we can resolve this by adding a qualification to the name
> > > (like
> > > > > > > Andrew
> > > > > > > > > > suggested, e.g., VECTOR_EMBEDDING or FINITE_VECTOR) to
> > avoid
> > > > > > > confusion
> > > > > > > > > with
> > > > > > > > > > broader ML use cases. Alternatively, we could introduce a
> > > > boolean
> > > > > > > flag
> > > > > > > > > > (e.g., optional bool VectorType.finite) as proposed
> > earlier.
> > > > > > > > > >
> > > > > > > > > > On Fri, Sep 25, 2026 at 3:52 AM Andrew Lamb <
> > > > > > [email protected]>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I am a little confused about the points of disagreement
> > > here.
> > > > > > > > > > >
> > > > > > > > > > > Is it that the name `VECTOR` is too general for the
> > > specific
> > > > > use
> > > > > > > case
> > > > > > > > > of
> > > > > > > > > > > fixed length arrays with non-finite values used for
> > vector
> > > > > > > > embeddings?
> > > > > > > > > > If
> > > > > > > > > > > so, maybe we could add a qualification to the name
> > > > > > (FINITE_VECTOR?
> > > > > > > > > > > FEATURE_VECTOR? VECTOR_EMBEDDING?)
> > > > > > > > > > >
> > > > > > > > > > > It seems like there is precedent for defining logical
> > types
> > > > > that
> > > > > > > > > specify
> > > > > > > > > > > certain physical types and restrict their contents to
> > help
> > > > > > readers
> > > > > > > > > > > interpret meaning, for example string[1] or JSON;
> > Defining
> > > > one
> > > > > > for
> > > > > > > > > > vector
> > > > > > > > > > > embeddings seems reasonable enough to me.
> > > > > > > > > > >
> > > > > > > > > > > Andrew
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > [1]:
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Sep 24, 2026 at 3:34 PM Daniel Weeks <
> > > > > [email protected]>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Well, Parquet is not a single-purpose format.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Vector stores can choose whatever policy fits their
> > > > usage,
> > > > > > but
> > > > > > > > > we're
> > > > > > > > > > > > > talking about a general-purpose file format that
> aims
> > > to
> > > > be
> > > > > > > > broadly
> > > > > > > > > > > > > applicable.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I agree that the format is intended to be general
> > > purpose,
> > > > > but
> > > > > > > > types
> > > > > > > > > > > within
> > > > > > > > > > > > the format define specific representation and
> semantics
> > > to
> > > > > > > achieve
> > > > > > > > > that
> > > > > > > > > > > > goal.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > "Vector" has a more general sense than ML and it's
> > > quite
> > > > > > > trivial
> > > > > > > > to
> > > > > > > > > > > > > produce vectors with non-finite values using e.g.
> > NumPy
> > > > or
> > > > > > > Pandas
> > > > > > > > > (or
> > > > > > > > > > > > > any other piece of dataframe or scientific
> computing
> > > > > > software).
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >  The justification for restricting non-finite values
> is
> > > > > > directly
> > > > > > > > > > related
> > > > > > > > > > > to
> > > > > > > > > > > > the semantic meaning of a vector.  Vectors have
> > specific
> > > > > > > > > > characteristics
> > > > > > > > > > > > (dimension, magnitude, direction, etc).  Non-finite
> > > values
> > > > > are
> > > > > > > > > invalid
> > > > > > > > > > > and
> > > > > > > > > > > > break those characteristics.  I'd liken it to storing
> > > > > non-utf8
> > > > > > > > values
> > > > > > > > > > in
> > > > > > > > > > > a
> > > > > > > > > > > > string and still calling it a string.
> > > > > > > > > > > >
> > > > > > > > > > > > That also differentiates a vector from a fixed length
> > > list,
> > > > > > which
> > > > > > > > is
> > > > > > > > > > more
> > > > > > > > > > > > general and does not carry the same semantic meaning.
> > > > > > > > > > > >
> > > > > > > > > > > > I also don't think we need to bundle all similarly
> > shaped
> > > > use
> > > > > > > cases
> > > > > > > > > > into
> > > > > > > > > > > > the same solution (e.g. flxed size list).  Vector is
> a
> > > well
> > > > > > > > > established
> > > > > > > > > > > > data type at this point and aligns with the general
> > > purpose
> > > > > > > nature
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > format (many other formats and databases natively
> > support
> > > > > > this).
> > > > > > > > > > > >
> > > > > > > > > > > > -Dan
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Sep 24, 2026 at 11:55 AM Antoine Pitrou <
> > > > > > > > [email protected]>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Le 24/09/2026 à 20:11, Russell Spitzer a écrit :
> > > > > > > > > > > > > > I think something I haven't seen yet is the clear
> > > > > use-case
> > > > > > > for
> > > > > > > > > > > > non-finite
> > > > > > > > > > > > > > values within a vector type. There has been a lot
> > of
> > > > > > > > theoretical
> > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > here about use cases but what toolings currently
> > > > produce
> > > > > > > these
> > > > > > > > > > > vectors
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > how do they function in downstream engines? I'd
> > like
> > > to
> > > > > > limit
> > > > > > > > > > > > introducing
> > > > > > > > > > > > > > types that have a wide range of values which are
> > > simply
> > > > > are
> > > > > > > > > failure
> > > > > > > > > > > > modes
> > > > > > > > > > > > > > of different flavors
> > > > > > > > > > > > >
> > > > > > > > > > > > > "Vector" has a more general sense than ML and it's
> > > quite
> > > > > > > trivial
> > > > > > > > to
> > > > > > > > > > > > > produce vectors with non-finite values using e.g.
> > NumPy
> > > > or
> > > > > > > Pandas
> > > > > > > > > (or
> > > > > > > > > > > > > any other piece of dataframe or scientific
> computing
> > > > > > software).
> > > > > > > > > > > > >
> > > > > > > > > > > > > It's not like the general concept of a vector is a
> > IETF
> > > > > > > standard
> > > > > > > > or
> > > > > > > > > > > > > anything, so analogies with JSON or UTF8 (where the
> > > > values
> > > > > > are
> > > > > > > > > > expected
> > > > > > > > > > > > > to be valid according to the respective specs) do
> not
> > > > > apply.
> > > > > > > > > > > > >
> > > > > > > > > > > > > This discussion would IMHO have a better chance of
> > > > > converging
> > > > > > > if
> > > > > > > > we
> > > > > > > > > > > > > separated the two orthogonal concerns of:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. representing fixed-length list/vector data in
> > > Parquet
> > > > > > (with
> > > > > > > a
> > > > > > > > > > > > > potential limitation to non-nested types)
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. representing the information that a given column
> > (or
> > > > > > column
> > > > > > > > > chunk,
> > > > > > > > > > > > > etc.) does not contain non-finite values
> > > > > > > > > > > > >
> > > > > > > > > > > > > ... both of which are useful on their own right,
> but
> > do
> > > > not
> > > > > > > have
> > > > > > > > to
> > > > > > > > > > be
> > > > > > > > > > > > > conflated and bundled together.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regards
> > > > > > > > > > > > >
> > > > > > > > > > > > > Antoine.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Reply via email to