Re: Existence/name/scope for minimal C/C++ Arrow C Data interface helpers

2022-07-07 Thread David Li
Now at https://github.com/apache/arrow-nanoarrow Dewey: you can use .asf.yaml to enable issues and such: https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-GitHubsettings On Thu, Jul 7, 2022, at 09:06, David Li wrote: > I'll go ahead and set up arrow-

Re: Existence/name/scope for minimal C/C++ Arrow C Data interface helpers

2022-07-07 Thread David Li
I'll go ahead and set up arrow-nanoarrow for convenience. In the medium term we should think about whether arrow-adbc and arrow-nanoarrow should be folded back into the arrow monorepo, in order to potentially reduce the release/CI maintenance burden, or document why we've chosen to split those

Re: Existence/name/scope for minimal C/C++ Arrow C Data interface helpers

2022-07-06 Thread Dewey Dunnington
I'm happy to develop anywhere anytime! My personal vote would be apache/arrow-nanoarrow because it highlights the minimal-ness of it but am happy to move forward however the community sees fit. Cheers, -dewey On Wed, Jul 6, 2022 at 12:46 PM Wes McKinney wrote: > hi all, > > Is there a path to

Re: Existence/name/scope for minimal C/C++ Arrow C Data interface helpers

2022-07-06 Thread Wes McKinney
hi all, Is there a path to doing this development work in project-owned repositories so the IP is "blessed" from an ASF governance / IP lineage standpoint? I see two potential routes: * Working in a subdirectory of apache/arrow * Creating a new repository like apache/arrow-c (or some other arrow-

Re: Existence/name/scope for minimal C/C++ Arrow C Data interface helpers

2022-06-25 Thread Dewey Dunnington
Hi all, Thanks for all the feedback so far! I've opened up two more draft PRs implementing [1] an API for owning buffers (precursor to creating struct ArrowArrays) and [2] an API for creating ArrowSchema objects for all Arrow types. All comments welcome! -dewey [1] https://github.com/paleolimbot

Re: Existence/name/scope for minimal C/C++ Arrow C Data interface helpers

2022-06-16 Thread Dewey Dunnington
> Can we name it miniarrow or nanoarrow? I'm happy to call it something else! Probably nanoarrow if I get to pick because of the parallel with nanopb/nanodbc. On Thu, Jun 16, 2022 at 6:26 AM Antoine Pitrou wrote: > > Can we name it miniarrow or nanoarrow? We don't want to convey the > message t

Re: Existence/name/scope for minimal C/C++ Arrow C Data interface helpers

2022-06-16 Thread Antoine Pitrou
Can we name it miniarrow or nanoarrow? We don't want to convey the message that there is a parallel C API for Arrow. Le 15/06/2022 à 05:18, Dewey Dunnington a écrit : Hi all, I drafted a second PR [1] drafting a design for storing parsed information obtained from a struct ArrowSchema (i.e.

Re: Existence/name/scope for minimal C/C++ Arrow C Data interface helpers

2022-06-14 Thread Dewey Dunnington
Hi all, I drafted a second PR [1] drafting a design for storing parsed information obtained from a struct ArrowSchema (i.e., parsing the format string into usable C structures). There are some unsolved problems that could use a fresh perspective...all comments welcome! [1] https://github.com/pale

Re: Existence/name/scope for minimal C/C++ Arrow C Data interface helpers

2022-06-10 Thread Dewey Dunnington
Hi all, As promised, I converted the design document [1] into an initial PR [2]. Rather than draft the whole header, I started with README + implementations + testing for error handling and schema allocation (depending on feedback, next week I will draft another reviewable chunk). Also feel free

Re: Existence/name/scope for minimal C/C++ Arrow C Data interface helpers

2022-06-03 Thread Dewey Dunnington
Hi all, Based on the points raised above and a few adventures implementing some of this in related projects, I put together a brief design document proposing a scope and structure to perhaps solidify a few of these discussions: https://docs.google.com/document/d/11n7ICVZO8exZ-z3GRlI26VLzKPXlYlEz5x

Re: Existence/name/scope for minimal C/C++ Arrow C Data interface helpers

2022-06-03 Thread Hannes Mühleisen
Hello List, we at DuckDB are happy users of the Arrow C Data Interface and use it to feed SQL queries and also use it to provide query results in Arrow format again. It is particularly appealing to us that the interface is merely a (C) header file that we just ship with our source code [1]. Intern

Re: Existence/name/scope for minimal C/C++ Arrow C Data interface helpers

2022-06-03 Thread Jonathan Keane
cc Hannes Mühleisen from DuckDB Labs -Jon On Tue, May 31, 2022 at 5:03 PM Wes McKinney wrote: > I'm also supportive of having a small vendorable C/C++ "Arrow > middleware" that provides: > > * Schemas and types > * Columnar data structures and minimal APIs to build them and iterate over > them

Re: Existence/name/scope for minimal C/C++ Arrow C Data interface helpers

2022-05-31 Thread Wes McKinney
I'm also supportive of having a small vendorable C/C++ "Arrow middleware" that provides: * Schemas and types * Columnar data structures and minimal APIs to build them and iterate over them * C data interface * Minimal validation (at the level of Validate but not ValidateFull) I don't think it's g

Re: Existence/name/scope for minimal C/C++ Arrow C Data interface helpers

2022-05-30 Thread Jonathan Keane
Thanks for working on this. I've heard people asking about something like this from a number of different fronts on top of the obvious use case in geoarrow | other geospatial libraries. I think a minimal piece of Arrow that other packages could depend on without needing to bring in all of arrow wou

Existence/name/scope for minimal C/C++ Arrow C Data interface helpers

2022-05-25 Thread Dewey Dunnington
I'm writing to gauge interest in a set of helpers in C and/or C++ for reading/exporting Arrow C Data interface structures. My use-case is building Arrow geospatial support in R [1], and while the set of helpers I've been using [2] has served the purpose of me writing about the opportunities for Arr