Hi All,
Currently the Rust implementation of arrow lacks a consistent story for
supporting scalars. Whilst there are some binary kernels that support
scalar values [1] [2], the way this is encoded is not consistent [3],
requires type-dispatch logic in downstreams like DataFusion [4], and has
no mechanism to preserve type metadata such as timestamp timezones, or
decimal precision [5].
I would therefore like to draw attention to a proposal [6] to address
this. As this will necessarily have downstream ramifications, and is
likely a problem that other implementations of arrow have already
grappled with, I would very much appreciate any feedback the community
can give. To avoid bifurcating the discussion, please comment on the
GitHub PR.
I look forward to hearing your thoughts.
Kind Regards,
Raphael Taylor-Davies
[1]:
https://docs.rs/arrow-arith/latest/arrow_arith/arithmetic/fn.add_scalar.html
[2]:
https://docs.rs/arrow-ord/latest/arrow_ord/comparison/fn.eq_dyn_scalar.html
[3]: https://github.com/apache/arrow-rs/issues/2837
[4]:
https://github.com/apache/arrow-datafusion/blob/d9e91d187c8af7f39999f8b1a11d53383826a471/datafusion/physical-expr/src/expressions/binary.rs
[5]: https://github.com/apache/arrow-rs/issues/3999
[6]: https://github.com/apache/arrow-rs/pull/4393
