I'm open to x-simd if others think it is the best option. I think the last time this came up I expressed this opinion, but if possible it would be nice to use something that is on its way to become a standard to avoid abandonment issues but I don't know enough about the space to understand if this is a real concern.
On Tue, Feb 9, 2021 at 7:00 PM Yuqi Gu <yuqi...@linaro.org> wrote: > Thanks for comments on the SIMD related PR: > https://github.com/apache/arrow/pull/9424. > Agree to adopt the *xsimd *as the SIMD wrapper library for Arrow to avoid a > large maintenance burden. It makes sense. > > It seems *ximd *is designed for mathematics calculating and it lacks the > functions like bit/byte shuffling, byte stream split encoding, ARM SVE > supporting, etc. > I'm absolutely willing to contribute the missing functions to *xsimd*. > > BRs, > Yuqi > > On Tue, 9 Feb 2021 at 22:08, Antoine Pitrou <anto...@python.org> wrote: > > > > > Le 09/02/2021 à 10:36, Antoine Pitrou a écrit : > > > > > > Note that we need to decouple the SIMD level available at compile-time > > > from the SIMD level available at runtime. That is, we typically build > > > optional AVX512 accelerations at compile-time, but only enable them at > > > runtime if the CPU supports AVX512 (and if the environment variable > > > ARROW_USER_SIMD_LEVEL wasn't forced to a lower value). > > > > > > From a quick glance, it's not obvious that xsimd supports that level of > > > control. Though it may just be undocumented. I will check with the > > > authors, since I happen to know them. > > > > Ok, there shouldn't be any problem on that front. We just need to > > compile with the right compiler flags to select the desired SIMD level, > > like we already do currently when compiling multiple versions of a > > function. > > > > I'll note that xsimd isn't very complete. For example, it seems to lack > > the functions required for byte stream split encoding and decoding. > > Those functions are exported by libsimdpp under the names "zip_lo" and > > "zip_hi". > > > > libsimdpp, on the other hand, seems to lack maintenance. It hasn't had > > a commit in one year, and issues and PRs seem to be unanswered. So > > perhaps xsimd is a better course, provided we want to contribute the > > missing functions. > > > > Regards > > > > Antoine. > > >
