Re: [RESULT] [VOTE] Alter Arrow binary protocol to address 8-byte Flatbuffer alignment requirements (2nd vote)

2019-09-06 Thread Ji Liu
Hi all, During the java code review[1], seems there is a problem with the current implementations(C++/Java etc) when reaching EOS, since the new format EOS is 8 bytes and the reader only reads 4 bytes when reach the end of stream, and the additional 4 bytes will not be read which cause

Re: [ANNOUNCE] New committers: Ben Kietzman, Kenta Murata, and Neal Richardson

2019-09-06 Thread Bryan Cutler
Congrats Ben, Kenta and Neal! On Fri, Sep 6, 2019, 12:15 PM Krisztián Szűcs wrote: > Congratulations! > > On Fri, Sep 6, 2019 at 8:12 PM Ben Kietzman > wrote: > > > Thanks! > > > > On Fri, Sep 6, 2019 at 1:09 PM Micah Kornfield > > wrote: > > > > > Congrats everyone! (apologies if I double

[jira] [Created] (ARROW-6480) [Developer] Add command to generate and send e-mail report for a Crossbow run

2019-09-06 Thread Wes McKinney (Jira)
Wes McKinney created ARROW-6480: --- Summary: [Developer] Add command to generate and send e-mail report for a Crossbow run Key: ARROW-6480 URL: https://issues.apache.org/jira/browse/ARROW-6480 Project:

Plasma scenarios

2019-09-06 Thread Eric Erhardt
I was looking for the high level scenarios for the Plasma In-Memory Object Store. A colleague of mine suggested we could use it to pass data between a C# process and a Python process. I've read the intro blog [0] on Plasma, which describes using the same data set from multiple processes -

Re: [ANNOUNCE] New committers: Ben Kietzman, Kenta Murata, and Neal Richardson

2019-09-06 Thread Krisztián Szűcs
Congratulations! On Fri, Sep 6, 2019 at 8:12 PM Ben Kietzman wrote: > Thanks! > > On Fri, Sep 6, 2019 at 1:09 PM Micah Kornfield > wrote: > > > Congrats everyone! (apologies if I double sent this). > > > > On Fri, Sep 6, 2019 at 10:06 AM Neal Richardson < > > neal.p.richard...@gmail.com> > >

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Krisztián Szűcs
On Fri, Sep 6, 2019 at 7:56 PM Wes McKinney wrote: > On Fri, Sep 6, 2019 at 3:18 AM Krisztián Szűcs > wrote: > > > > Hey Wes, > > > > On Fri, Sep 6, 2019 at 12:23 AM Wes McKinney > wrote: > > > > > hi Krisztian, > > > > > > Anyone who's developing in the project can see that the Buildbot setup

Re: [ANNOUNCE] New committers: Ben Kietzman, Kenta Murata, and Neal Richardson

2019-09-06 Thread Ben Kietzman
Thanks! On Fri, Sep 6, 2019 at 1:09 PM Micah Kornfield wrote: > Congrats everyone! (apologies if I double sent this). > > On Fri, Sep 6, 2019 at 10:06 AM Neal Richardson < > neal.p.richard...@gmail.com> > wrote: > > > Thanks, y'all! > > > > On Fri, Sep 6, 2019 at 5:44 AM David Li wrote: > > >

Re: [DISCUSS] IPC buffer layout for Null type

2019-09-06 Thread Wes McKinney
On Fri, Sep 6, 2019 at 12:57 PM Micah Kornfield wrote: > > > > > We can't because the buffer layout is not transmitted -- implementations > > make assumptions about what Buffer values correspond to each field. The > > only thing we could do to signal the change would be to increase the > >

Re: [DISCUSS] IPC buffer layout for Null type

2019-09-06 Thread Micah Kornfield
> > We can't because the buffer layout is not transmitted -- implementations > make assumptions about what Buffer values correspond to each field. The > only thing we could do to signal the change would be to increase the > metadata version from V4 to V5. If we do this within 0.15.0 we could

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Wes McKinney
On Fri, Sep 6, 2019 at 3:18 AM Krisztián Szűcs wrote: > > Hey Wes, > > On Fri, Sep 6, 2019 at 12:23 AM Wes McKinney wrote: > > > hi Krisztian, > > > > Anyone who's developing in the project can see that the Buildbot setup > > is working well (at least for Linux builds) and giving much more > >

Re: [ANNOUNCE] New committers: Ben Kietzman, Kenta Murata, and Neal Richardson

2019-09-06 Thread Micah Kornfield
Congrats everyone! (apologies if I double sent this). On Fri, Sep 6, 2019 at 10:06 AM Neal Richardson wrote: > Thanks, y'all! > > On Fri, Sep 6, 2019 at 5:44 AM David Li wrote: > > > > Congrats all! :) > > > > Best, > > David > > > > On 9/6/19, Francois Saint-Jacques wrote: > > > Congrats to

Re: [DISCUSS] IPC buffer layout for Null type

2019-09-06 Thread Antoine Pitrou
Null can also come up when converting a column with only NA values in a CSV file. I don't remember for sure, but I think the same can happen with JSON files as well. Can't we accept both forms when reading? It sounds like it should be reasonably easy. Regards Antoine. Le 06/09/2019 à

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Antoine Pitrou
Le 06/09/2019 à 16:18, Krisztián Szűcs a écrit : > On Fri, Sep 6, 2019 at 12:48 PM Antoine Pitrou wrote: > >> On Fri, 6 Sep 2019 12:41:15 +0200 >> Krisztián Szűcs wrote: I get the impression that it is a complicated and fragile solution to the problem. >>> Ursabot has a

Re: [ANNOUNCE] New committers: Ben Kietzman, Kenta Murata, and Neal Richardson

2019-09-06 Thread Neal Richardson
Thanks, y'all! On Fri, Sep 6, 2019 at 5:44 AM David Li wrote: > > Congrats all! :) > > Best, > David > > On 9/6/19, Francois Saint-Jacques wrote: > > Congrats to everyone! > > > > François > > > > On Fri, Sep 6, 2019 at 4:34 AM Kenta Murata wrote: > >> > >> Thank you very much everyone! > >>

Re: [DISCUSS][FORMAT] Concerning about character encoding of binary string data

2019-09-06 Thread Kenta Murata
Thanks for responding. I understand ExtensionType is suitable for handling character encoding. I'll try to make and propose draft specification and implementation of an extension type. Regards, Kenta Murata 2019年9月5日(木) 7:56 Wes McKinney : > > I opened

[jira] [Created] (ARROW-6479) [C++] inline errors from external projects' build logs

2019-09-06 Thread Benjamin Kietzman (Jira)
Benjamin Kietzman created ARROW-6479: Summary: [C++] inline errors from external projects' build logs Key: ARROW-6479 URL: https://issues.apache.org/jira/browse/ARROW-6479 Project: Apache Arrow

[jira] [Created] (ARROW-6478) [C++] Roll back to jemalloc stable-4 branch until performance issues in 5.2.x addressed

2019-09-06 Thread Wes McKinney (Jira)
Wes McKinney created ARROW-6478: --- Summary: [C++] Roll back to jemalloc stable-4 branch until performance issues in 5.2.x addressed Key: ARROW-6478 URL: https://issues.apache.org/jira/browse/ARROW-6478

Re: [DISCUSS] IPC buffer layout for Null type

2019-09-06 Thread Wes McKinney
hi Micah, Null wouldn't come up that often in practice. It could happen when converting from pandas, for example In [8]: df = pd.DataFrame({'col1': np.array([np.nan] * 10, dtype=object)}) In [9]: t = pa.table(df) In [10]: t Out[10]: pyarrow.Table col1: null metadata {b'pandas':

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Krisztián Szűcs
On Fri, Sep 6, 2019 at 12:48 PM Antoine Pitrou wrote: > On Fri, 6 Sep 2019 12:41:15 +0200 > Krisztián Szűcs wrote: > > > > > > I get the impression that it is a complicated and fragile solution to > > > the problem. > > > > > Ursabot has a bunch of tests to ensure that we don't brake any of the

[jira] [Created] (ARROW-6477) [Packaging][Crossbow] Use Azure Pipelines to build linux packages

2019-09-06 Thread Krisztian Szucs (Jira)
Krisztian Szucs created ARROW-6477: -- Summary: [Packaging][Crossbow] Use Azure Pipelines to build linux packages Key: ARROW-6477 URL: https://issues.apache.org/jira/browse/ARROW-6477 Project: Apache

Re: [ANNOUNCE] New committers: Ben Kietzman, Kenta Murata, and Neal Richardson

2019-09-06 Thread David Li
Congrats all! :) Best, David On 9/6/19, Francois Saint-Jacques wrote: > Congrats to everyone! > > François > > On Fri, Sep 6, 2019 at 4:34 AM Kenta Murata wrote: >> >> Thank you very much everyone! >> I'm very happy to join this community. >> >> 2019年9月6日(金) 12:39 Micah Kornfield : >> >> > >>

[jira] [Created] (ARROW-6476) [Java][CI] Travis java all-jdks job is broken

2019-09-06 Thread Francois Saint-Jacques (Jira)
Francois Saint-Jacques created ARROW-6476: - Summary: [Java][CI] Travis java all-jdks job is broken Key: ARROW-6476 URL: https://issues.apache.org/jira/browse/ARROW-6476 Project: Apache Arrow

Re: [ANNOUNCE] New committers: Ben Kietzman, Kenta Murata, and Neal Richardson

2019-09-06 Thread Francois Saint-Jacques
Congrats to everyone! François On Fri, Sep 6, 2019 at 4:34 AM Kenta Murata wrote: > > Thank you very much everyone! > I'm very happy to join this community. > > 2019年9月6日(金) 12:39 Micah Kornfield : > > > > > Congrats everyone. > > > > On Thu, Sep 5, 2019 at 7:06 PM Ji Liu wrote: > > > > >

[jira] [Created] (ARROW-6475) [C++] Don't try to dictionary encode dictionary arrays

2019-09-06 Thread Krisztian Szucs (Jira)
Krisztian Szucs created ARROW-6475: -- Summary: [C++] Don't try to dictionary encode dictionary arrays Key: ARROW-6475 URL: https://issues.apache.org/jira/browse/ARROW-6475 Project: Apache Arrow

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Antoine Pitrou
On Fri, 6 Sep 2019 12:41:15 +0200 Krisztián Szűcs wrote: > > > > I get the impression that it is a complicated and fragile solution to > > the problem. > > > Ursabot has a bunch of tests to ensure that we don't brake any of the > functionality, > so fragility can be avoided by testing it.

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Krisztián Szűcs
On Fri, Sep 6, 2019 at 12:15 PM Antoine Pitrou wrote: > > Le 06/09/2019 à 12:13, Krisztián Szűcs a écrit : > > On Fri, Sep 6, 2019 at 12:01 PM Antoine Pitrou > wrote: > > > >> > >> Le 06/09/2019 à 10:07, Krisztián Szűcs a écrit : > >>> For example trigger a builder for changes affecting files

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Antoine Pitrou
Le 06/09/2019 à 12:13, Krisztián Szűcs a écrit : > On Fri, Sep 6, 2019 at 12:01 PM Antoine Pitrou wrote: > >> >> Le 06/09/2019 à 10:07, Krisztián Szűcs a écrit : >>> For example trigger a builder for changes affecting files under arrow/ci >>> which reloads the builder object within the build

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Krisztián Szűcs
On Fri, Sep 6, 2019 at 12:01 PM Antoine Pitrou wrote: > > Le 06/09/2019 à 10:07, Krisztián Szűcs a écrit : > > For example trigger a builder for changes affecting files under arrow/ci > > which reloads the builder object within the build master's process. > > I am asking you how this affects

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Antoine Pitrou
Le 06/09/2019 à 10:07, Krisztián Szűcs a écrit : > For example trigger a builder for changes affecting files under arrow/ci > which reloads the builder object within the build master's process. I am asking you how this affects only the current build and not other concurrent builds. Regards

Re: [ANNOUNCE] New committers: Ben Kietzman, Kenta Murata, and Neal Richardson

2019-09-06 Thread Kenta Murata
Thank you very much everyone! I'm very happy to join this community. 2019年9月6日(金) 12:39 Micah Kornfield : > > Congrats everyone. > > On Thu, Sep 5, 2019 at 7:06 PM Ji Liu wrote: > > > Congratulations! > > > > Thanks, > > Ji Liu > > > > > >

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Krisztián Szűcs
On Fri, Sep 6, 2019 at 12:23 AM Wes McKinney wrote: > hi Krisztian, > > Anyone who's developing in the project can see that the Buildbot setup > is working well (at least for Linux builds) and giving much more > timely feedback, which has been very helpful. > > I'm concerned about the "ursabot"

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Krisztián Szűcs
Hey Wes, On Fri, Sep 6, 2019 at 12:23 AM Wes McKinney wrote: > hi Krisztian, > > Anyone who's developing in the project can see that the Buildbot setup > is working well (at least for Linux builds) and giving much more > timely feedback, which has been very helpful. > > I'm concerned about the

Re: [PROPOSAL] Consolidate Arrow's CI configuration

2019-09-06 Thread Krisztián Szűcs
For example trigger a builder for changes affecting files under arrow/ci which reloads the builder object within the build master's process. We are not limited to shell commands, arbitrary python functions can be executed too, but the semantics would be similar to running MasterShellCommand [1].