Re: [DISCUSS] Plasma appears to have been forked, consider deprecating pyarrow.serialization

2020-07-12 Thread Wes McKinney
I'll add deprecation warnings to the pyarrow.serialize functions in question, it will be pretty simple. On Sun, Jul 12, 2020, 6:34 PM Neal Richardson wrote: > This seems like something to investigate after the 1.0 release. > > Neal > > On Sun, Jul 12, 2020 at 11:53 AM Antoine Pitrou > wrote: >

Re: [DISCUSS] Plasma appears to have been forked, consider deprecating pyarrow.serialization

2020-07-12 Thread Neal Richardson
This seems like something to investigate after the 1.0 release. Neal On Sun, Jul 12, 2020 at 11:53 AM Antoine Pitrou wrote: > > I'd certainly like to deprecate our custom Python serialization format, > and using pickle protocol 5 instead is a very good idea. > > We can probably keep it in 1.0 w

Re: [DISCUSS] Ongoing LZ4 problems with Parquet files

2020-07-12 Thread Wes McKinney
Since there hasn't been other movement on this, we need to disable writing LZ4-compressed files until this can be investigated more thoroughly. If someone wants to submit a patch that would be helpful otherwise I can take a look in the next couple days On Thu, Jul 2, 2020 at 12:50 PM Antoine Pitro

Re: [EXTERNAL] Re: .NET support for Arrow

2020-07-12 Thread anthony . abate
Wes, >I find that people have many reasons for not contributing to >an existing open source project, so I want to make sure I know what >yours are, whether one of: For the record, I have contributed to this project: both PRs and Jiras for various 'bugs' as I found them - some (most?) made their w

Re: [EXTERNAL] Re: .NET support for Arrow

2020-07-12 Thread Wes McKinney
On Sun, Jul 12, 2020 at 2:44 PM wrote: > > Wes, > > I thought Arrow was (or at least includes) an open standard for > interoperability? There are even specific 'implementation guidelines' > regarding supporting parts or all of the specification. That's true, but at the moment there is not any C#

Re: [EXTERNAL] Re: .NET support for Arrow

2020-07-12 Thread Adam Szmigin
Hi Anthony, On 12/07/2020 20:43, anthony.ab...@gmail.com wrote: It appears that fragmentation is already a problem (ie private forks) I should point out that the private fork currently used by my organisation is a "minimal divergence" from the upstream project, and it was my intention from th

Re: [EXTERNAL] Re: .NET support for Arrow

2020-07-12 Thread anthony . abate
Wes, I thought Arrow was (or at least includes) an open standard for interoperability? There are even specific 'implementation guidelines' regarding supporting parts or all of the specification. It appears that fragmentation is already a problem (ie private forks) Where I work, we don't trust th

Re: [DISCUSS] Plasma appears to have been forked, consider deprecating pyarrow.serialization

2020-07-12 Thread Antoine Pitrou
I'd certainly like to deprecate our custom Python serialization format, and using pickle protocol 5 instead is a very good idea. We can probably keep it in 1.0 while raising a FutureWarning. Regards Antoine. Le 12/07/2020 à 19:22, Wes McKinney a écrit : > It appears that the Ray developers h

Re: [EXTERNAL] Re: .NET support for Arrow

2020-07-12 Thread Wes McKinney
hi Anthony, On Sun, Jul 12, 2020 at 12:13 PM wrote: > > I am in the same position as Adam - We don't use the official apache arrow > library any more either and have been using an old fork with our own > (probably the same) bug fixes. > > Personally, I have somewhat given up on the Apache .Net li

[DISCUSS] Plasma appears to have been forked, consider deprecating pyarrow.serialization

2020-07-12 Thread Wes McKinney
It appears that the Ray developers have decided to fork Plasma and decouple from the Arrow codebase: https://github.com/ray-project/ray/pull/9154 This is a disappointing development to occur without any discussion on this mailing list but given the lack of development activity on Plasma I would l

Re: [EXTERNAL] Re: .NET support for Arrow

2020-07-12 Thread anthony . abate
I am in the same position as Adam - We don't use the official apache arrow library any more either and have been using an old fork with our own (probably the same) bug fixes. Personally, I have somewhat given up on the Apache .Net library... I have an alternative C# arrow library that I have writt

Re: Property-driven Parquet encryption

2020-07-12 Thread Gidon Gershinsky
Hi Micah, Thanks for your comments here, and at the design googledoc. We'll get started, we've got the input we were looking for. Cheers, Gidon

[NIGHTLY] Arrow Build Report for Job nightly-2020-07-12-0

2020-07-12 Thread Crossbow
Arrow Build Report for Job nightly-2020-07-12-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-07-12-0 Failed Tasks: - gandiva-jar-xenial: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-07-12-0-travis-gandiva-jar-xenial - test-co