[Rust][DataFusion] Supporting input_file_name()

2021-02-23 Thread Mike Seddon
Hi, One of Apache Spark's very useful SQL functions is the 'input_file_name' SQL function which provides a simple API for identifying the source of a row of data when sourced from a file-based source like Parquet or CSV. This is particularly useful for identifying which chunk/partition of a

Re: DataFusion Postgres License Requirements

2021-02-23 Thread Sutou Kouhei
Hi, Could someone (including not Rust developers) take a look at this? C++ mentions third-party source explicitly. For example, https://github.com/apache/arrow/blob/master/cpp/src/arrow/status.h mentions that arrow::Status is based on codes from LevelDB, Apache Kudu and TensorFlow. DataFusion

Re: [C++][CI] Make "C++ on s390x" build mandatory?

2021-02-23 Thread Bryan Cutler
+1 sgtm On Tue, Feb 23, 2021, 9:47 AM Micah Kornfield wrote: > +1, but let's keep an eye on it to make sure it remains stable. > > On Tue, Feb 23, 2021 at 5:34 AM Kazuaki Ishizaki > wrote: > > > Thank you. +1 for this proposal, > > > > Kazuaki Ishizaki > > > > > > > > From: Benjamin Kietzman

RE: [DataFusion] Promoting GroupByScalar to public API

2021-02-23 Thread Ruan Pearce-Authers
Hi Daniël, That makes sense! Especially from a performance perspective for future and revised aggregate implementations. I'll keep my copy-pasted GroupByScalar implementation for my prototype work (where perf isn't super critical yet) and try to get involved in any future discussions on this

Re: [DataFusion] Promoting GroupByScalar to public API

2021-02-23 Thread Daniël Heres
Hi Ruan, I am not sure about any stability guidelines, I didn't hear of it. Aside from this, I think there might be some arguments against making GroupByScalar public for a number of reasons: * We have both ScalarValue and GroupByValue, with some duplication/mapping between them. I think it

Re: [C++] libarrow isolation

2021-02-23 Thread Antoine Pitrou
Hi Tahsin, I see. So the error happens when loading PyArrow into MATLAB, I suppose? What kind of error do you get? Regards Antoine. Le 23/02/2021 à 18:12, Tahsin Hassan a écrit : > Hi Antoine, > > MATLAB is using RTLD_GLOBAL. Hope that helps in clarifying the workflow. > > Regrards, >

Re: [C++][CI] Make "C++ on s390x" build mandatory?

2021-02-23 Thread Micah Kornfield
+1, but let's keep an eye on it to make sure it remains stable. On Tue, Feb 23, 2021 at 5:34 AM Kazuaki Ishizaki wrote: > Thank you. +1 for this proposal, > > Kazuaki Ishizaki > > > > From: Benjamin Kietzman > To: dev > Date: 2021/02/23 21:19 > Subject:[EXTERNAL] Re: [C++][CI]

[DataFusion] Promoting GroupByScalar to public API

2021-02-23 Thread Ruan Pearce-Authers
Hey all, Whilst working on some UDAFs, I noticed I essentially had to reimplement GroupByScalar to use scalars as HashMap keys inside accumulator struct state, as ScalarValue (correctly!) doesn't implement Eq/Hash. A simple fix to ease this process would be to remove the crate-only access

Re: [C++] libarrow isolation

2021-02-23 Thread Tahsin Hassan
Hi Antoine, MATLAB is using RTLD_GLOBAL. Hope that helps in clarifying the workflow. Regrards, Tahsin From: Antoine Pitrou Sent: Monday, February 22, 2021 9:41 AM To: dev@arrow.apache.org Subject: Re: [C++] libarrow isolation Le 22/02/2021 à 15:29, Tahsin

Re: [C++][CI] Make "C++ on s390x" build mandatory?

2021-02-23 Thread Kazuaki Ishizaki
Thank you. +1 for this proposal, Kazuaki Ishizaki From: Benjamin Kietzman To: dev Date: 2021/02/23 21:19 Subject:[EXTERNAL] Re: [C++][CI] Make "C++ on s390x" build mandatory? +1 for making it mandatory On Tue, Feb 23, 2021, 07:07 Krisztián Szűcs wrote: > Hi! > > On

Re: [C++][CI] Make "C++ on s390x" build mandatory?

2021-02-23 Thread Benjamin Kietzman
+1 for making it mandatory On Tue, Feb 23, 2021, 07:07 Krisztián Szűcs wrote: > Hi! > > On Tue, Feb 23, 2021 at 11:53 AM Antoine Pitrou > wrote: > > > > > > Hello, > > > > For a while we've had a big endian (s390x-based) build on Travis-CI. > > The build is optional, meaning errors don't

Re: [C++][CI] Make "C++ on s390x" build mandatory?

2021-02-23 Thread Krisztián Szűcs
Hi! On Tue, Feb 23, 2021 at 11:53 AM Antoine Pitrou wrote: > > > Hello, > > For a while we've had a big endian (s390x-based) build on Travis-CI. > The build is optional, meaning errors don't actually fail the CI. > > The build has been reasonably stable for some time apart for some > occasional

[C++][CI] Make "C++ on s390x" build mandatory?

2021-02-23 Thread Antoine Pitrou
Hello, For a while we've had a big endian (s390x-based) build on Travis-CI. The build is optional, meaning errors don't actually fail the CI. The build has been reasonably stable for some time apart for some occasional regressions, which often don't get spotted because the build is reported as

[NIGHTLY] Arrow Build Report for Job nightly-2021-02-23-0

2021-02-23 Thread Crossbow
Arrow Build Report for Job nightly-2021-02-23-0 All tasks: https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-02-23-0 Failed Tasks: - conda-linux-gcc-py36-aarch64: URL: