[JAVA] Supporting zero copy arrow-vector

2018-09-06 Thread Zhenyuan Zhao
Hello Team, I'm working on using arrow as intermediate format for transferring columnar data from server to client. In this case, the client will only need to read from the format so I would like to avoid any unnecessary copy of the data. Looking into arrow, while arrow-format/flatbuffers does sup

[jira] [Created] (ARROW-3186) [GLib] mesonbuild failures in Travis CI

2018-09-06 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-3186: --- Summary: [GLib] mesonbuild failures in Travis CI Key: ARROW-3186 URL: https://issues.apache.org/jira/browse/ARROW-3186 Project: Apache Arrow Issue Type: Bug

[jira] [Created] (ARROW-3185) [C++] Address libparquet SO version convention in unified build

2018-09-06 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-3185: --- Summary: [C++] Address libparquet SO version convention in unified build Key: ARROW-3185 URL: https://issues.apache.org/jira/browse/ARROW-3185 Project: Apache Arrow

Re: Assembling an initial development roadmap for Arrow in R

2018-09-06 Thread Wes McKinney
I'm going to transfer the document over to Confluence soon and start creating JIRA issues. If anyone has further comments about the roadmap, please add them to the document or request edit access. Thanks! On Tue, Aug 28, 2018 at 5:16 PM Wes McKinney wrote: > > hi folks, > > In light of Romain's p

[jira] [Created] (ARROW-3184) [C++] Add modular build targets, "all" target, and require explicit target when invoking make or ninja

2018-09-06 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-3184: --- Summary: [C++] Add modular build targets, "all" target, and require explicit target when invoking make or ninja Key: ARROW-3184 URL: https://issues.apache.org/jira/browse/ARROW-3184

[jira] [Created] (ARROW-3183) get_library_dirs on Windows gives the wrong directory

2018-09-06 Thread Victor Uriarte (JIRA)
Victor Uriarte created ARROW-3183: - Summary: get_library_dirs on Windows gives the wrong directory Key: ARROW-3183 URL: https://issues.apache.org/jira/browse/ARROW-3183 Project: Apache Arrow

[jira] [Created] (ARROW-3182) [C++] Merge Gandiva codebase

2018-09-06 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-3182: --- Summary: [C++] Merge Gandiva codebase Key: ARROW-3182 URL: https://issues.apache.org/jira/browse/ARROW-3182 Project: Apache Arrow Issue Type: New Feature

Re: Lighter build matrix on a language specific fork.

2018-09-06 Thread Romain François
Thanks. I will do that first thing in the morning. I just skimmed through them and it’s nothing dramatic, mostly oversights. I just can’t act on them right now, #parenting 👨‍👧 > Le 7 sept. 2018 à 00:40, Wes McKinney a écrit : > > OK, if you could address the comments that I left, after that I

Re: Lighter build matrix on a language specific fork.

2018-09-06 Thread Wes McKinney
OK, if you could address the comments that I left, after that I can merge the PR On Thu, Sep 6, 2018 at 8:39 AM Romain François wrote: > As far as I’m concerned the initial pr is good to go, the intent is to > just have an r package that builds against the C++ library and that checks > on travis

Re: Lighter build matrix on a language specific fork.

2018-09-06 Thread Romain François
As far as I’m concerned the initial pr is good to go, the intent is to just have an r package that builds against the C++ library and that checks on travis. Actual code that does stuff will follow. (I have two branches on top of it for later). But this is bare minimal by design. Romain >

[jira] [Created] (ARROW-3181) [Packaging] Adjust conda package scripts to account for Parquet codebase migration

2018-09-06 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-3181: --- Summary: [Packaging] Adjust conda package scripts to account for Parquet codebase migration Key: ARROW-3181 URL: https://issues.apache.org/jira/browse/ARROW-3181 Projec

[jira] [Created] (ARROW-3180) [C++] Add docker-compose setup to simulate Travis CI run locally

2018-09-06 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-3180: --- Summary: [C++] Add docker-compose setup to simulate Travis CI run locally Key: ARROW-3180 URL: https://issues.apache.org/jira/browse/ARROW-3180 Project: Apache Arrow

Re: Lighter build matrix on a language specific fork.

2018-09-06 Thread Wes McKinney
Yes, as soon as the initial R PR is in (and the CI scripts aren't changing) the build will be faster. @Romain how much more work do you want to do on the initial PR? We can review and merge by end of week if that sounds good On Thu, Sep 6, 2018 at 6:10 AM Uwe L. Korn wrote: > The problem could

Re: Lighter build matrix on a language specific fork.

2018-09-06 Thread Uwe L. Korn
The problem could be that it checks against master and you will probably have changes for R in the ci/ directory. Changes in that directory will trigger a build for the full matrix. So to get the build simple and fast, we should get the ci/ changes for R into master soon. Uwe On Thu, Sep 6, 20

Re: Lighter build matrix on a language specific fork.

2018-09-06 Thread Antoine Pitrou
Le 06/09/2018 à 15:03, Romain François a écrit : > I must do something wrong then because it builds them all, all the time 🤷‍♂️. Can you show an example? > > Not a big deal, it’s only about 10 jobs, and i really only care about the r > job and the only doing the 🐀 analysis. > > Commenting

Re: Lighter build matrix on a language specific fork.

2018-09-06 Thread Romain François
I must do something wrong then because it builds them all, all the time 🤷‍♂️. Not a big deal, it’s only about 10 jobs, and i really only care about the r job and the only doing the 🐀 analysis. Commenting out may not be very practical as the plan is to submit small pull requests, so it’s almost

Re: Lighter build matrix on a language specific fork.

2018-09-06 Thread Antoine Pitrou
Our CI harness will already fast-exit in jobs that are not affected by the current changes (if you change only the R directory, C++ jobs will exit early). If you want it to be even faster, your best bet is to temporarily comment out job entries in .travis.yml. Regards Antoine. Le 06/09/2018

Lighter build matrix on a language specific fork.

2018-09-06 Thread Romain François
Hello, Is there a way to have a lighter build matrix on travis, perhaps based on the branch name, for example when working on the r bindings and not touching anything else, having only the r job to be triggered would make it faster for travis. For example when working on r features i would t

[jira] [Created] (ARROW-3179) [Python] Move all the test data files to apache/arrow-testing

2018-09-06 Thread Krisztian Szucs (JIRA)
Krisztian Szucs created ARROW-3179: -- Summary: [Python] Move all the test data files to apache/arrow-testing Key: ARROW-3179 URL: https://issues.apache.org/jira/browse/ARROW-3179 Project: Apache Arrow

Re: [DISCUSS] Dropping support for CentOS 5 / RHEL5 in Python packages

2018-09-06 Thread Uwe L. Korn
Hello Wes, I'm ok with option 2 when we use the yet unfinished manylinux2010 image as the base. This way, we will still be able to produce wheels that in the near future are actually based an a architecture tag supported by a PEP. Also as I have some packaging nightmare, I would feel much bette