Re: Rust/Datafusion sort kernel issues

2020-09-02 Thread Matthias Vallentin
Would it perhaps make sense to define the total order for non-numbers (NaN, Inf, -Inf) globally (i.e., in the spec or in Arrow itself) so that the behavior is the same across all languages? On Fri, Aug 28, 2020 at 7:42 PM Andy Grove wrote: > Hi Jörn, > > I agree with your concerns about NaN.

Re: Rust/Datafusion sort kernel issues

2020-08-28 Thread Andy Grove
Hi Jörn, I agree with your concerns about NaN. There was a discussion about this in https://github.com/apache/arrow/pull/7193 I will try and make time this weekend to look at the current implementation and your suggestions around DictionaryArray. Hopefully, other contributors that are more

Rust/Datafusion sort kernel issues

2020-08-28 Thread Jörn Horstmann
I ran into a few issues with the rust sort kernels and would like to discuss possible solutions. 1. When sorting by multiple columns (lexsort_to_indices) the Float32 and Float64 data types are not supported because the implementation relies on the OrdArray trait. This trait is not implemented