Le 07/06/2024 à 04:27, Felipe Oliveira Carvalho a écrit :
I've been thinking about how to encode statistics on Arrow arrays and
how to keep the set of statistics known by both producers and
consumers (i.e. standardized).
The statistics array(s) could be a
map<
// the column index or null if the statistics refer to whole table or batch
column: int32,
map<int32, dense_union<...needed types based on stat kinds in the keys...>>
>
The keys would be defined as part of the standard:
// Statistics values are identified by specified int32-valued keys
// so that producers and consumers can agree on physical
// encoding and semantics. Statistics can be about a column,
// a record batch, or both.
typedef ArrowStatKind int32_t;
One thing that a plain integer makes more difficult is representing
non-standard statistics. For example some engine might want to expose
elaborate quantile-based statistics even if it not officially defined
here. With a `utf8` or `dictionary(int32, utf8)` field, that is quite
easy with some prefixing to ensure uniqueness. With a `int32` field, the
spec would have to mention a mechanism to ensure global uniqueness of
vendor-specific statistics.
Version markers in two-sided protocols never work well long term:
see Parquet files lying about the version of the encoder so the files
can be read and web browsers lying on their User-Agent strings so
websites don't break. It's better to allow probing for individual
feature support (in this case, the presence of a specific stat kind in
the array).
+1 on this.
Regards
Antoine.
