Re: [Discuss] [Rust] Arrow2/parquet2 going foward

2021-07-18 Thread Jorge Cardoso Leitão
Hi, Whatever its technical faults may be, projects that rely on arrow (such as > anything based on DataFusion, like my own) need to be supported as they > have made the bet on Rust Arrow. > 1.X versioning in Apache Arrow was never meant to represent stability of their individual libraries, but on

Re: [DISCUSS][C++] Strategies for SIMD cross-compilation?

2021-07-18 Thread Yuqi Gu
*> So rather than using xsimd::batch for an AVX512 batch,>you would do xsimd::batch (or e.g.>neon/neon64 for ARM ISAs) and then access the batch size through the>batch::size static property.* Glad to see xsimd use 'Arch' as the parameter of a 'batch'. For the ARROW-11502

Re: [DISCUSS][C++] Strategies for SIMD cross-compilation?

2021-07-18 Thread Yibo Cai
On 7/17/21 12:08 AM, Wes McKinney wrote: hi folks, I had a conversation with the developers of xsimd last week in Paris and was made aware that they are working on a substantial refactor of xsimd to improve its usability for cross-compilation and dynamic-dispatch based on runtime processor ca

Re: [Discuss] [Rust] Arrow2/parquet2 going foward

2021-07-18 Thread Adam Lippai
Hi Simon, There are several Arrow implementations in parallel: https://arrow.apache.org/docs/status.html The Python and R versions are based on Arrow C++, others are completely separate projects. Arrow-rs and Arrow2 are referring to the Rust implementation, Arrow C++ is not going to be replaced.

Re: [Discuss] [Rust] Arrow2/parquet2 going foward

2021-07-18 Thread Simon Perkins
I'm interested to hear what the relation between arrow2, arrow-rs and the main github apache/arrow is. Is the intention to replace the C++ codebase with a rust implementation? The reason I'm asking is that I'm adding complex number support in the C++ codebase. It may instead be a better idea to do

Re: [Rust] [DataFusion] Proposal for datafusion test reorganization

2021-07-18 Thread Andrew Lamb
I finally had a chance to write this up: https://github.com/apache/arrow-datafusion/issues/743 Sorry that took so long -- most of my open source time recently has been focused on the arrow release process Andrew On Thu, Apr 15, 2021 at 7:05 AM Andrew Lamb wrote: > Thanks Daniël, > > I'll write