Re: [C++] Runtime SIMD dispatching for Arrow

2020-09-06 Thread Wes McKinney
I might be able to contribute an AVX-512 capable machine for testing / benchmarking via Buildkite or similar in the next 6 months. It seems like dedicated hardware would be the best approach to get consistency there. If someone else would be able to contribute a reliable machine that would also be

Re: Adding Parquet encryption support to PyArrow

2020-09-06 Thread Itamar Turner-Trauring
On Tuesday when I'm back at work I will read all the above, and can coordinate on starting a design doc. On Sun, Sep 6, 2020, at 5:03 AM, Gidon Gershinsky wrote: > Cool, thank you. This would solve the problem at hand. > I agree it'd be good to kick off the PyArrow API discussion in parallel >

[NIGHTLY] Arrow Build Report for Job nightly-2020-09-06-0

2020-09-06 Thread Crossbow
Arrow Build Report for Job nightly-2020-09-06-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-09-06-0 Failed Tasks: - centos-8-aarch64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-09-06-0-travis-centos-8-aarch64 -

Re: Adding Parquet encryption support to PyArrow

2020-09-06 Thread Gidon Gershinsky
Cool, thank you. This would solve the problem at hand. I agree it'd be good to kick off the PyArrow API discussion in parallel with the PR8023 review. Maybe you and Itamar could prep a googledoc draft for the community to have a look and to comment. Cheers, Gidon On Fri, Sep 4, 2020 at 6:08 PM

Re: Adding Parquet encryption support to PyArrow

2020-09-06 Thread Gidon Gershinsky
Here it goes, as promised. Briefly, the low level interface is harder to use, demands deeper encryption expertise, and requires a developer to manually implement a number of components, critical for production deployment. The high level interface is simple and has these critical components