Re: [RESULT] [VOTE] Alter Arrow binary protocol to address 8-byte Flatbuffer alignment requirements (2nd vote)

Hi all, During the java code review[1], seems there is a problem with the current implementations(C++/Java etc) when reaching EOS, since the new format EOS is 8 bytes and the reader only reads 4 bytes when reach the end of stream, and the additional 4 bytes will not be read which cause

Re: [ANNOUNCE] New committers: Ben Kietzman, Kenta Murata, and Neal Richardson

Congrats Ben, Kenta and Neal! On Fri, Sep 6, 2019, 12:15 PM Krisztián Szűcs wrote: > Congratulations! > > On Fri, Sep 6, 2019 at 8:12 PM Ben Kietzman > wrote: > > > Thanks! > > > > On Fri, Sep 6, 2019 at 1:09 PM Micah Kornfield > > wrote: > > > > > Congrats everyone! (apologies if I double

[jira] [Created] (ARROW-6480) [Developer] Add command to generate and send e-mail report for a Crossbow run

Wes McKinney created ARROW-6480: --- Summary: [Developer] Add command to generate and send e-mail report for a Crossbow run Key: ARROW-6480 URL: https://issues.apache.org/jira/browse/ARROW-6480 Project:

Plasma scenarios

I was looking for the high level scenarios for the Plasma In-Memory Object Store. A colleague of mine suggested we could use it to pass data between a C# process and a Python process. I've read the intro blog [0] on Plasma, which describes using the same data set from multiple processes -

Re: [ANNOUNCE] New committers: Ben Kietzman, Kenta Murata, and Neal Richardson

Congratulations! On Fri, Sep 6, 2019 at 8:12 PM Ben Kietzman wrote: > Thanks! > > On Fri, Sep 6, 2019 at 1:09 PM Micah Kornfield > wrote: > > > Congrats everyone! (apologies if I double sent this). > > > > On Fri, Sep 6, 2019 at 10:06 AM Neal Richardson < > > neal.p.richard...@gmail.com> > >

Re: [PROPOSAL] Consolidate Arrow's CI configuration

On Fri, Sep 6, 2019 at 7:56 PM Wes McKinney wrote: > On Fri, Sep 6, 2019 at 3:18 AM Krisztián Szűcs > wrote: > > > > Hey Wes, > > > > On Fri, Sep 6, 2019 at 12:23 AM Wes McKinney > wrote: > > > > > hi Krisztian, > > > > > > Anyone who's developing in the project can see that the Buildbot setup

Re: [ANNOUNCE] New committers: Ben Kietzman, Kenta Murata, and Neal Richardson

Thanks! On Fri, Sep 6, 2019 at 1:09 PM Micah Kornfield wrote: > Congrats everyone! (apologies if I double sent this). > > On Fri, Sep 6, 2019 at 10:06 AM Neal Richardson < > neal.p.richard...@gmail.com> > wrote: > > > Thanks, y'all! > > > > On Fri, Sep 6, 2019 at 5:44 AM David Li wrote: > > >

Re: [DISCUSS] IPC buffer layout for Null type

On Fri, Sep 6, 2019 at 12:57 PM Micah Kornfield wrote: > > > > > We can't because the buffer layout is not transmitted -- implementations > > make assumptions about what Buffer values correspond to each field. The > > only thing we could do to signal the change would be to increase the > >

Re: [DISCUSS] IPC buffer layout for Null type

> > We can't because the buffer layout is not transmitted -- implementations > make assumptions about what Buffer values correspond to each field. The > only thing we could do to signal the change would be to increase the > metadata version from V4 to V5. If we do this within 0.15.0 we could

Re: [PROPOSAL] Consolidate Arrow's CI configuration

On Fri, Sep 6, 2019 at 3:18 AM Krisztián Szűcs wrote: > > Hey Wes, > > On Fri, Sep 6, 2019 at 12:23 AM Wes McKinney wrote: > > > hi Krisztian, > > > > Anyone who's developing in the project can see that the Buildbot setup > > is working well (at least for Linux builds) and giving much more > >

Re: [ANNOUNCE] New committers: Ben Kietzman, Kenta Murata, and Neal Richardson

Congrats everyone! (apologies if I double sent this). On Fri, Sep 6, 2019 at 10:06 AM Neal Richardson wrote: > Thanks, y'all! > > On Fri, Sep 6, 2019 at 5:44 AM David Li wrote: > > > > Congrats all! :) > > > > Best, > > David > > > > On 9/6/19, Francois Saint-Jacques wrote: > > > Congrats to

Re: [DISCUSS] IPC buffer layout for Null type

Null can also come up when converting a column with only NA values in a CSV file. I don't remember for sure, but I think the same can happen with JSON files as well. Can't we accept both forms when reading? It sounds like it should be reasonably easy. Regards Antoine. Le 06/09/2019 à

Re: [PROPOSAL] Consolidate Arrow's CI configuration

Le 06/09/2019 à 16:18, Krisztián Szűcs a écrit : > On Fri, Sep 6, 2019 at 12:48 PM Antoine Pitrou wrote: > >> On Fri, 6 Sep 2019 12:41:15 +0200 >> Krisztián Szűcs wrote: I get the impression that it is a complicated and fragile solution to the problem. >>> Ursabot has a

Re: [ANNOUNCE] New committers: Ben Kietzman, Kenta Murata, and Neal Richardson

Thanks, y'all! On Fri, Sep 6, 2019 at 5:44 AM David Li wrote: > > Congrats all! :) > > Best, > David > > On 9/6/19, Francois Saint-Jacques wrote: > > Congrats to everyone! > > > > François > > > > On Fri, Sep 6, 2019 at 4:34 AM Kenta Murata wrote: > >> > >> Thank you very much everyone! > >>

Re: [DISCUSS][FORMAT] Concerning about character encoding of binary string data

Thanks for responding. I understand ExtensionType is suitable for handling character encoding. I'll try to make and propose draft specification and implementation of an extension type. Regards, Kenta Murata 2019年9月5日(木) 7:56 Wes McKinney : > > I opened

[jira] [Created] (ARROW-6479) [C++] inline errors from external projects' build logs

Benjamin Kietzman created ARROW-6479: Summary: [C++] inline errors from external projects' build logs Key: ARROW-6479 URL: https://issues.apache.org/jira/browse/ARROW-6479 Project: Apache Arrow

[jira] [Created] (ARROW-6478) [C++] Roll back to jemalloc stable-4 branch until performance issues in 5.2.x addressed

Wes McKinney created ARROW-6478: --- Summary: [C++] Roll back to jemalloc stable-4 branch until performance issues in 5.2.x addressed Key: ARROW-6478 URL: https://issues.apache.org/jira/browse/ARROW-6478

Re: [DISCUSS] IPC buffer layout for Null type

hi Micah, Null wouldn't come up that often in practice. It could happen when converting from pandas, for example In [8]: df = pd.DataFrame({'col1': np.array([np.nan] * 10, dtype=object)}) In [9]: t = pa.table(df) In [10]: t Out[10]: pyarrow.Table col1: null metadata {b'pandas':

Re: [PROPOSAL] Consolidate Arrow's CI configuration

On Fri, Sep 6, 2019 at 12:48 PM Antoine Pitrou wrote: > On Fri, 6 Sep 2019 12:41:15 +0200 > Krisztián Szűcs wrote: > > > > > > I get the impression that it is a complicated and fragile solution to > > > the problem. > > > > > Ursabot has a bunch of tests to ensure that we don't brake any of the

[jira] [Created] (ARROW-6477) [Packaging][Crossbow] Use Azure Pipelines to build linux packages

Krisztian Szucs created ARROW-6477: -- Summary: [Packaging][Crossbow] Use Azure Pipelines to build linux packages Key: ARROW-6477 URL: https://issues.apache.org/jira/browse/ARROW-6477 Project: Apache

Re: [ANNOUNCE] New committers: Ben Kietzman, Kenta Murata, and Neal Richardson

Congrats all! :) Best, David On 9/6/19, Francois Saint-Jacques wrote: > Congrats to everyone! > > François > > On Fri, Sep 6, 2019 at 4:34 AM Kenta Murata wrote: >> >> Thank you very much everyone! >> I'm very happy to join this community. >> >> 2019年9月6日(金) 12:39 Micah Kornfield : >> >> > >>

[jira] [Created] (ARROW-6476) [Java][CI] Travis java all-jdks job is broken

Francois Saint-Jacques created ARROW-6476: - Summary: [Java][CI] Travis java all-jdks job is broken Key: ARROW-6476 URL: https://issues.apache.org/jira/browse/ARROW-6476 Project: Apache Arrow

Re: [ANNOUNCE] New committers: Ben Kietzman, Kenta Murata, and Neal Richardson

Congrats to everyone! François On Fri, Sep 6, 2019 at 4:34 AM Kenta Murata wrote: > > Thank you very much everyone! > I'm very happy to join this community. > > 2019年9月6日(金) 12:39 Micah Kornfield : > > > > > Congrats everyone. > > > > On Thu, Sep 5, 2019 at 7:06 PM Ji Liu wrote: > > > > >

[jira] [Created] (ARROW-6475) [C++] Don't try to dictionary encode dictionary arrays

Krisztian Szucs created ARROW-6475: -- Summary: [C++] Don't try to dictionary encode dictionary arrays Key: ARROW-6475 URL: https://issues.apache.org/jira/browse/ARROW-6475 Project: Apache Arrow

Re: [PROPOSAL] Consolidate Arrow's CI configuration

On Fri, 6 Sep 2019 12:41:15 +0200 Krisztián Szűcs wrote: > > > > I get the impression that it is a complicated and fragile solution to > > the problem. > > > Ursabot has a bunch of tests to ensure that we don't brake any of the > functionality, > so fragility can be avoided by testing it.

Re: [PROPOSAL] Consolidate Arrow's CI configuration

On Fri, Sep 6, 2019 at 12:15 PM Antoine Pitrou wrote: > > Le 06/09/2019 à 12:13, Krisztián Szűcs a écrit : > > On Fri, Sep 6, 2019 at 12:01 PM Antoine Pitrou > wrote: > > > >> > >> Le 06/09/2019 à 10:07, Krisztián Szűcs a écrit : > >>> For example trigger a builder for changes affecting files

Re: [PROPOSAL] Consolidate Arrow's CI configuration

Le 06/09/2019 à 12:13, Krisztián Szűcs a écrit : > On Fri, Sep 6, 2019 at 12:01 PM Antoine Pitrou wrote: > >> >> Le 06/09/2019 à 10:07, Krisztián Szűcs a écrit : >>> For example trigger a builder for changes affecting files under arrow/ci >>> which reloads the builder object within the build

Re: [PROPOSAL] Consolidate Arrow's CI configuration

On Fri, Sep 6, 2019 at 12:01 PM Antoine Pitrou wrote: > > Le 06/09/2019 à 10:07, Krisztián Szűcs a écrit : > > For example trigger a builder for changes affecting files under arrow/ci > > which reloads the builder object within the build master's process. > > I am asking you how this affects

Re: [PROPOSAL] Consolidate Arrow's CI configuration

Le 06/09/2019 à 10:07, Krisztián Szűcs a écrit : > For example trigger a builder for changes affecting files under arrow/ci > which reloads the builder object within the build master's process. I am asking you how this affects only the current build and not other concurrent builds. Regards

Re: [ANNOUNCE] New committers: Ben Kietzman, Kenta Murata, and Neal Richardson

Thank you very much everyone! I'm very happy to join this community. 2019年9月6日(金) 12:39 Micah Kornfield : > > Congrats everyone. > > On Thu, Sep 5, 2019 at 7:06 PM Ji Liu wrote: > > > Congratulations! > > > > Thanks, > > Ji Liu > > > > > >

Re: [PROPOSAL] Consolidate Arrow's CI configuration

On Fri, Sep 6, 2019 at 12:23 AM Wes McKinney wrote: > hi Krisztian, > > Anyone who's developing in the project can see that the Buildbot setup > is working well (at least for Linux builds) and giving much more > timely feedback, which has been very helpful. > > I'm concerned about the "ursabot"

Re: [PROPOSAL] Consolidate Arrow's CI configuration

Hey Wes, On Fri, Sep 6, 2019 at 12:23 AM Wes McKinney wrote: > hi Krisztian, > > Anyone who's developing in the project can see that the Buildbot setup > is working well (at least for Linux builds) and giving much more > timely feedback, which has been very helpful. > > I'm concerned about the

Re: [PROPOSAL] Consolidate Arrow's CI configuration

For example trigger a builder for changes affecting files under arrow/ci which reloads the builder object within the build master's process. We are not limited to shell commands, arbitrary python functions can be executed too, but the semantics would be similar to running MasterShellCommand [1].