[jira] [Created] (ARROW-5860) [Java] [Vector] Fix decimal byte setter

2019-07-04 Thread Praveen Kumar Desabandu (JIRA)
Praveen Kumar Desabandu created ARROW-5860: -- Summary: [Java] [Vector] Fix decimal byte setter Key: ARROW-5860 URL: https://issues.apache.org/jira/browse/ARROW-5860 Project: Apache Arrow

[Discuss][Java] Make the semantics of lastSet consistent

2019-07-04 Thread Fan Liya
There are two lastSet member variables in the code. One is in BaseVariableWidthVector and the other is in ListVector. In BaseVariableWidthVector, the lastSet refers to the last index that is actually set, while in ListVector, the lastSet refers to the next index that will be set. So there is an inc

Re: linking 3rd party cython modules against pyarrow fails since 0.14.0

2019-07-04 Thread Stestagg
1) pip install pyarrow==0.14.0 2) All the pyarrow files including, for example libarrow.so.14, but not libarrow.so (hence the linker error) Reproducible on Python 3.7.2 on linux mint 19.1 and debian docker: Example dockerfile: ``` FROM debian:unstable-slim RUN apt-get update && apt-get upgrade -

[jira] [Created] (ARROW-5859) [Python] Support ExtentionType on conversion to numpy/pandas

2019-07-04 Thread Joris Van den Bossche (JIRA)
Joris Van den Bossche created ARROW-5859: Summary: [Python] Support ExtentionType on conversion to numpy/pandas Key: ARROW-5859 URL: https://issues.apache.org/jira/browse/ARROW-5859 Project: A

[jira] [Created] (ARROW-5858) [Doc] Better document the Tensor classes in the prose documentation

2019-07-04 Thread Joris Van den Bossche (JIRA)
Joris Van den Bossche created ARROW-5858: Summary: [Doc] Better document the Tensor classes in the prose documentation Key: ARROW-5858 URL: https://issues.apache.org/jira/browse/ARROW-5858 Pro

Re: linking 3rd party cython modules against pyarrow fails since 0.14.0

2019-07-04 Thread Antoine Pitrou
Hi, 1) How did you install PyArrow? 2) What does /usr/local/lib/python3.7/dist-packages/pyarrow contain? Regards Antoine. Le 04/07/2019 à 22:10, Stestagg a écrit : > Hi > > I've got a cython module that links against PyArrow, using the > 'pyarrow.get_libraries()' associated methods. > > B

[jira] [Created] (ARROW-5857) [Python] converting multidimensional numpy arrays to nested list type

2019-07-04 Thread Joris Van den Bossche (JIRA)
Joris Van den Bossche created ARROW-5857: Summary: [Python] converting multidimensional numpy arrays to nested list type Key: ARROW-5857 URL: https://issues.apache.org/jira/browse/ARROW-5857 P

[jira] [Created] (ARROW-5856) linking 3rd party cython modules against pyarrow fails since 0.14.0

2019-07-04 Thread Steve Stagg (JIRA)
Steve Stagg created ARROW-5856: -- Summary: linking 3rd party cython modules against pyarrow fails since 0.14.0 Key: ARROW-5856 URL: https://issues.apache.org/jira/browse/ARROW-5856 Project: Apache Arrow

linking 3rd party cython modules against pyarrow fails since 0.14.0

2019-07-04 Thread Stestagg
Hi I've got a cython module that links against PyArrow, using the 'pyarrow.get_libraries()' associated methods. Builds on Windows and Linux are consitently failing against 0.14, but working on 0.12 to 0.13. Linux gives: x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-

[jira] [Created] (ARROW-5855) [Python] Add support for Duration type

2019-07-04 Thread Joris Van den Bossche (JIRA)
Joris Van den Bossche created ARROW-5855: Summary: [Python] Add support for Duration type Key: ARROW-5855 URL: https://issues.apache.org/jira/browse/ARROW-5855 Project: Apache Arrow I

[jira] [Created] (ARROW-5854) [Python] Expose compare kernels on Array class

2019-07-04 Thread Joris Van den Bossche (JIRA)
Joris Van den Bossche created ARROW-5854: Summary: [Python] Expose compare kernels on Array class Key: ARROW-5854 URL: https://issues.apache.org/jira/browse/ARROW-5854 Project: Apache Arrow

[jira] [Created] (ARROW-5853) [Python] Expose boolean filter kernel on Array

2019-07-04 Thread Joris Van den Bossche (JIRA)
Joris Van den Bossche created ARROW-5853: Summary: [Python] Expose boolean filter kernel on Array Key: ARROW-5853 URL: https://issues.apache.org/jira/browse/ARROW-5853 Project: Apache Arrow

Re: Flight authentication interoperability

2019-07-04 Thread Ryan Murray
Hey David, I was actually testing test_flight.test_http_basic_auth(). But I think the same applies. The Java default implementation expects a handshake. More to the point it expects a BasicAuth protobuf which I believe is not exposed at all in python. Always returning true in BasicServerAuthHandle

Re: Flight authentication interoperability

2019-07-04 Thread David Li
Hmm, interesting. I assume you mean test_flight.test_token_auth() as the client? The tests weren't written to be explicitly compatible, but there's no reason you should get an indefinite stall. We don't use Handshake/ServerAuthHandler#authenticate, so that would explain why we don't see issues. I

[CI] Ursabot Java builders

2019-07-04 Thread Krisztián Szűcs
Hi, I've added simple Java builders for JDK 8 and 11. The required change was quite small [1], but a build takes a bit more than 2 minutes [2]. Adding Go and Rust builders should be similarly easy. Regard, Krisztian [1]: https://github.com/ursa-labs/ursabot/commit/476a8b07fd81f6d7664dc5e115e079d

Re: Flight authentication interoperability

2019-07-04 Thread Ryan Murray
Hey David, I am curious to see what you are doing different from me. I am running the Java ExampleFlightServer.java against the python auth flight tests and they are not passing. The particular issue is that incoming.next() never returns in BasicServerAuthHandler.java:56 It doesn't appear to be a

[jira] [Created] (ARROW-5852) [Rust] [DataFusion] Compilation error with Rust 1.37.0-nightly

2019-07-04 Thread Andy Grove (JIRA)
Andy Grove created ARROW-5852: - Summary: [Rust] [DataFusion] Compilation error with Rust 1.37.0-nightly Key: ARROW-5852 URL: https://issues.apache.org/jira/browse/ARROW-5852 Project: Apache Arrow

Re: Flight authentication interoperability

2019-07-04 Thread Antoine Pitrou
It may be worth opening a JIRA for the flaky tests if not already done. Regards Antoine. Le 04/07/2019 à 18:11, David Li a écrit : > I'm also curious as to what the issue was, as we've been doing > Python-client-Java-server auth with development builds without > trouble. > > Regardless - thi

Re: Flight authentication interoperability

2019-07-04 Thread David Li
I'm also curious as to what the issue was, as we've been doing Python-client-Java-server auth with development builds without trouble. Regardless - this does point out a need for more cross-language Flight testing (perhaps a Flight-specific integration suite?), and to get existing tests running mo

[jira] [Created] (ARROW-5851) [C++] Compilation of reference benchmarks fails

2019-07-04 Thread Antoine Pitrou (JIRA)
Antoine Pitrou created ARROW-5851: - Summary: [C++] Compilation of reference benchmarks fails Key: ARROW-5851 URL: https://issues.apache.org/jira/browse/ARROW-5851 Project: Apache Arrow Issue

Re: Release verification process

2019-07-04 Thread Wes McKinney
hi Neal, On Thu, Jul 4, 2019 at 9:31 AM Neal Richardson wrote: > > Hi all, > [Splitting off from the [VOTE] discussion thread] > > I wonder if there are things we can do to make for a smoother release next > time. This was the first release I participated in with the Arrow project, > and there we

Re: [VOTE] Release Apache Arrow 0.14.0 - RC0

2019-07-04 Thread Wes McKinney
On Thu, Jul 4, 2019 at 9:25 AM Jacques Nadeau wrote: > > On Thu, Jul 4, 2019, 7:05 AM Wes McKinney wrote: > > > hi Jacques, > > > > I agree that we should allow ample time for discussions around issues > > raised by people validating the release. > > > > That is the main point here. There should

[jira] [Created] (ARROW-5850) [CI][R] R appveyor job is broken after release

2019-07-04 Thread Neal Richardson (JIRA)
Neal Richardson created ARROW-5850: -- Summary: [CI][R] R appveyor job is broken after release Key: ARROW-5850 URL: https://issues.apache.org/jira/browse/ARROW-5850 Project: Apache Arrow Issue

Re: Release standards

2019-07-04 Thread Jacques Nadeau
You don't hold the release for the issue, you hold it for consensus. This allows other people to come to the same conclusion (or not). That's why it's called community over code. On Thu, Jul 4, 2019, 7:33 AM Antoine Pitrou wrote: > On Thu, 4 Jul 2019 07:25:05 -0700 > Jacques Nadeau wrote: > > >

Re: Flight authentication interoperability

2019-07-04 Thread Jacques Nadeau
Which is exactly why I was withholding a vote until there was more information. On Thu, Jul 4, 2019, 7:25 AM Antoine Pitrou wrote: > On Thu, 4 Jul 2019 09:04:34 -0500 > Wes McKinney wrote: > > > > That being said, with Ryan's issue, he is using a feature > > (cross-language auth in Flight) that

Re: Release standards

2019-07-04 Thread Antoine Pitrou
On Thu, 4 Jul 2019 07:25:05 -0700 Jacques Nadeau wrote: > > > I'd be glad to help make a point release in the near future since we > > uncovered quite a few small issues during this RC, and I think there's > > more value in getting the software into the hands of users than > > waiting multiple we

Release verification process

2019-07-04 Thread Neal Richardson
Hi all, [Splitting off from the [VOTE] discussion thread] I wonder if there are things we can do to make for a smoother release next time. This was the first release I participated in with the Arrow project, and there were some aspects that were surprising to me. It seemed odd that "release verifi

Re: [VOTE] Release Apache Arrow 0.14.0 - RC0

2019-07-04 Thread Jacques Nadeau
On Thu, Jul 4, 2019, 7:05 AM Wes McKinney wrote: > hi Jacques, > > I agree that we should allow ample time for discussions around issues > raised by people validating the release. > That is the main point here. There should have been a discussion about this. Different people are expecting differ

Re: Flight authentication interoperability

2019-07-04 Thread Antoine Pitrou
On Thu, 4 Jul 2019 09:04:34 -0500 Wes McKinney wrote: > > That being said, with Ryan's issue, he is using a feature > (cross-language auth in Flight) that isn't being tested. The Flight > integration tests do not use authentication AFAIK so I'm not surprised > to hear that there may be an issue w

Re: [VOTE] Release Apache Arrow 0.14.0 - RC0

2019-07-04 Thread Wes McKinney
hi Jacques, I agree that we should allow ample time for discussions around issues raised by people validating the release. That being said, with Ryan's issue, he is using a feature (cross-language auth in Flight) that isn't being tested. The Flight integration tests do not use authentication AFAI

Re: [VOTE] Release Apache Arrow 0.14.0 - RC0

2019-07-04 Thread Jacques Nadeau
A release vote should last until we arrive at consensus. When an issue is potentially identified, those that have voted should be given ample time to change their vote and others that may have been lazy consenters should be given time to chime in. There is no maximum amount of time a vote can be op

Re: [VOTE] Release Apache Arrow 0.14.0 - RC0

2019-07-04 Thread Antoine Pitrou
That's an open question. How long should a release vote last? Apparently this one lasted 3 days. The 0.14.0 vote (after RC4) lasted 3 days before a decision was made too. Regards Antoine. Le 04/07/2019 à 15:28, Jacques Nadeau a écrit : > There are two different questions here. > > 1) shoul

Re: [RESULT][VOTE] Release Apache Arrow 0.14.0 - RC0

2019-07-04 Thread Neal Richardson
I'll get the ball rolling on the R packages. Neal On Thu, Jul 4, 2019 at 12:46 AM Sutou Kouhei wrote: > Hi, > > I need your help! > Could Rust developers see "Failed:" section? > Could someone take over tasks in "Need helped:" section? > > Failed: > > * Updating Rust packages > > https://cwik

Re: [VOTE] Release Apache Arrow 0.14.0 - RC0

2019-07-04 Thread Jacques Nadeau
There are two different questions here. 1) should this issue block the release. 2) should a discussion with ample time for all parties to come to consensus have occurred before closing the vote. The first question is a subjective question. The answer to the second question should have been an une

Re: [VOTE] Release Apache Arrow 0.14.0 - RC0

2019-07-04 Thread Krisztián Szűcs
The issue report was hardly reproducible, and it wasn't captured by any of the automatized test suites. Until we don't have automatized tests for a feature I wouldn't consider it stable. If this is something we should have blocked the release on, then we can still draft a minor release after fixing

Re: [VOTE] Release Apache Arrow 0.14.0 - RC0

2019-07-04 Thread Antoine Pitrou
I agree with Kou here. It's not a problem with the release per se, it's a problem with the Flight code in git master. There is no regression AFAICT, and we have not promised that Flight was stable and production-ready yet. If we're ok releasing with bugs such as this (Java unable to read back

Re: [VOTE] Release Apache Arrow 0.14.0 - RC0

2019-07-04 Thread Jacques Nadeau
Im disappointed in this response. When someone finds an issue with a release it should be triaged/validated rather than rushed past rather quickly and closing the vote before others can chime in on the issue. I'm now left voting -1 on a closed vote. In the future, let's have a discussion about iss

[jira] [Created] (ARROW-5849) Compiler warnings on mingw-w64

2019-07-04 Thread Jeroen (JIRA)
Jeroen created ARROW-5849: - Summary: Compiler warnings on mingw-w64 Key: ARROW-5849 URL: https://issues.apache.org/jira/browse/ARROW-5849 Project: Apache Arrow Issue Type: Bug Components: C

Re: [DISCUSS] C++ SO versioning with 1.0.0

2019-07-04 Thread Krisztián Szűcs
Created a PR with the versioning scheme described by @Kou. https://github.com/apache/arrow/pull/4801 We can change later (but before the next release) to unblock the failing CI builds. On Thu, Jul 4, 2019 at 12:51 PM Antoine Pitrou wrote: > On Thu, 4 Jul 2019 09:50:10 +0200 > Krisztián Szűcs w

[jira] [Created] (ARROW-5848) [C++] SO versioning schema after release 1.0.0

2019-07-04 Thread Krisztian Szucs (JIRA)
Krisztian Szucs created ARROW-5848: -- Summary: [C++] SO versioning schema after release 1.0.0 Key: ARROW-5848 URL: https://issues.apache.org/jira/browse/ARROW-5848 Project: Apache Arrow Issue

[jira] [Created] (ARROW-5847) [C++] Remove lazy.h

2019-07-04 Thread Antoine Pitrou (JIRA)
Antoine Pitrou created ARROW-5847: - Summary: [C++] Remove lazy.h Key: ARROW-5847 URL: https://issues.apache.org/jira/browse/ARROW-5847 Project: Apache Arrow Issue Type: Task Compone

Re: [DISCUSS] C++ SO versioning with 1.0.0

2019-07-04 Thread Antoine Pitrou
On Thu, 4 Jul 2019 09:50:10 +0200 Krisztián Szűcs wrote: > Could We add a temporary solution and perhaps change the versioning schema > later? > I'm asking because all of our c++ based CI builds are failing now. Wow, thanks for pointing out. Indeed: https://travis-ci.org/apache/arrow/builds/55411

[jira] [Created] (ARROW-5846) [Java] Create Avro adapter module and add dependencies

2019-07-04 Thread Ji Liu (JIRA)
Ji Liu created ARROW-5846: - Summary: [Java] Create Avro adapter module and add dependencies Key: ARROW-5846 URL: https://issues.apache.org/jira/browse/ARROW-5846 Project: Apache Arrow Issue Type: Sub

[jira] [Created] (ARROW-5845) [Java] Implement converter between Arrow record batches and Avro records

2019-07-04 Thread Ji Liu (JIRA)
Ji Liu created ARROW-5845: - Summary: [Java] Implement converter between Arrow record batches and Avro records Key: ARROW-5845 URL: https://issues.apache.org/jira/browse/ARROW-5845 Project: Apache Arrow

[jira] [Created] (ARROW-5844) [Java] Support comparison & sort for more numeric types

2019-07-04 Thread Liya Fan (JIRA)
Liya Fan created ARROW-5844: --- Summary: [Java] Support comparison & sort for more numeric types Key: ARROW-5844 URL: https://issues.apache.org/jira/browse/ARROW-5844 Project: Apache Arrow Issue Type

[jira] [Created] (ARROW-5843) [Java] Improve the readability and performance of BitVectorHelper#getNullCount

2019-07-04 Thread Liya Fan (JIRA)
Liya Fan created ARROW-5843: --- Summary: [Java] Improve the readability and performance of BitVectorHelper#getNullCount Key: ARROW-5843 URL: https://issues.apache.org/jira/browse/ARROW-5843 Project: Apache Ar

Re: [RESULT][VOTE] Release Apache Arrow 0.14.0 - RC0

2019-07-04 Thread Sutou Kouhei
Thanks! Please see my note in other e-mail: >* We need to add Flight support >* We need to remove -D*_HOME and use system libraries as > much as possible In <95846f16-84c6-4cbe-8fe1-b16e73faf...@gmail.com> "Re: [RESULT][VOTE] Release Apache Arrow 0.14.0 - RC0" on Thu, 4 Jul 2019

Re: [RESULT][VOTE] Release Apache Arrow 0.14.0 - RC0

2019-07-04 Thread Sutou Kouhei
Thanks! In "Re: [RESULT][VOTE] Release Apache Arrow 0.14.0 - RC0" on Thu, 4 Jul 2019 09:32:55 +0200, Krisztián Szűcs wrote: > I will handle the conda-forge packages. > > On Thu, Jul 4, 2019 at 7:49 AM Sutou Kouhei wrote: > >> Hi, >> >> The 0.14.0 RC0 vote carries with 4 binding +1 votes

Re: [DISCUSS] C++ SO versioning with 1.0.0

2019-07-04 Thread Krisztián Szűcs
Could We add a temporary solution and perhaps change the versioning schema later? I'm asking because all of our c++ based CI builds are failing now. On Wed, Jul 3, 2019 at 10:51 PM Antoine Pitrou wrote: > > The question: is do we want to maintain a stable ABI? > > Given the number of constraints

Re: [RESULT][VOTE] Release Apache Arrow 0.14.0 - RC0

2019-07-04 Thread Sutou Kouhei
Hi, I need your help! Could Rust developers see "Failed:" section? Could someone take over tasks in "Need helped:" section? Failed: * Updating Rust packages https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide#ReleaseManagementGuide-UpdatingRustpackages * We nee

Re: [RESULT][VOTE] Release Apache Arrow 0.14.0 - RC0

2019-07-04 Thread Yosuke Shiro
I'll update Homebrew packages. > On Jul 4, 2019, at 14:49, Sutou Kouhei wrote: > > Hi, > > The 0.14.0 RC0 vote carries with 4 binding +1 votes. Thanks > for helping verify the RC. > > There are some post release tasks: > > > https://cwiki.apache.org/confluence/display/ARROW/Release+Managem

[jira] [Created] (ARROW-5842) [Java] Revise the semantic of lastSet in ListVector

2019-07-04 Thread Liya Fan (JIRA)
Liya Fan created ARROW-5842: --- Summary: [Java] Revise the semantic of lastSet in ListVector Key: ARROW-5842 URL: https://issues.apache.org/jira/browse/ARROW-5842 Project: Apache Arrow Issue Type: Bu

Re: [RESULT][VOTE] Release Apache Arrow 0.14.0 - RC0

2019-07-04 Thread Krisztián Szűcs
I will handle the conda-forge packages. On Thu, Jul 4, 2019 at 7:49 AM Sutou Kouhei wrote: > Hi, > > The 0.14.0 RC0 vote carries with 4 binding +1 votes. Thanks > for helping verify the RC. > > There are some post release tasks: > > > https://cwiki.apache.org/confluence/display/ARROW/Release+Man

[jira] [Created] (ARROW-5841) [Website] Add 0.14.0 release note

2019-07-04 Thread Sutou Kouhei (JIRA)
Sutou Kouhei created ARROW-5841: --- Summary: [Website] Add 0.14.0 release note Key: ARROW-5841 URL: https://issues.apache.org/jira/browse/ARROW-5841 Project: Apache Arrow Issue Type: Task