Re: [DISCUSS] Adding a pull-style iterator API to the C data interface

2020-08-25 Thread Antoine Pitrou
I proposed an API here: https://github.com/apache/arrow/pull/8052 It is not much different from what Wes proposed earlier in the thread, except in error reporting. Comments welcome (here or on the PR). Regards Antoine. Le 16/08/2020 à 21:28, Wes McKinney a écrit : > I opened

Re: [DISCUSS] Adding a pull-style iterator API to the C data interface

2020-08-16 Thread Wes McKinney
I opened https://issues.apache.org/jira/browse/ARROW-9761 about adding a preliminary C++ (and Python) implementation to help stir the pot. My understanding is that DuckDB is working on using the C interface right now [1] and the absence of an iterator interface makes such integration require more

Re: [DISCUSS] Adding a pull-style iterator API to the C data interface

2020-08-14 Thread Jacques Nadeau
I think this unlocks a bunch of use cases. I think people are generally using Arrow in simpler, non-streaming ways right now and thus the quiet. Producing an iterator pattern is logical as you move to streams of smaller chunks (common in distributed and multi-tenant systems). On Mon, Aug 10, 2020

Re: [DISCUSS] Adding a pull-style iterator API to the C data interface

2020-08-10 Thread Wes McKinney
I'm still in need of it. I'd be interested in developing a solution that can be used in some database APIs, e.g. using it for the result interface for an embedded SQL database like SQLite or DuckDB would be an interesting motivating use case. One approach would be to create something unofficial

Re: [DISCUSS] Adding a pull-style iterator API to the C data interface

2020-08-10 Thread Antoine Pitrou
From the absence of response, it would seem there isn't much interest in this. Please speak up if you think this would be useful to you. Regards Antoine. On Tue, 7 Jul 2020 07:49:17 -0500 Wes McKinney wrote: > Any opinions about this? It seems the next steps would be a concrete > API

Re: [DISCUSS] Adding a pull-style iterator API to the C data interface

2020-07-07 Thread Wes McKinney
Any opinions about this? It seems the next steps would be a concrete API proposal and perhaps a reference implementation thereof. On Sun, Jun 28, 2020 at 11:26 PM Wes McKinney wrote: > > In ARROW-8301 [1] and elsewhere we've been discussing how to > communicate what amounts to a sequence of

[DISCUSS] Adding a pull-style iterator API to the C data interface

2020-06-28 Thread Wes McKinney
In ARROW-8301 [1] and elsewhere we've been discussing how to communicate what amounts to a sequence of arrays or a sequence of RecordBatch objects using the C data interface. Example use cases: * Returning a sequence of record / row batches from a database driver * Sending a C++