Re: user-defined Python-based data-sources in Arrow

2022-06-24 Thread Weston Pace
s mot since > no parallel-access occurs. OTOH, if the Python-based data-source can be > accessed in parallel, the above sorting-queue solution is better suited and > would avoid the reentrancy problem of a ReadNext function. > > > Yaron. > ___

Re: user-defined Python-based data-sources in Arrow

2022-06-24 Thread Yaron Gvili
e the data-source function as producing > schema-carrying tabular data. > > > Yaron. > > From: Li Jin > Sent: Wednesday, June 22, 2022 2:50 PM > To: dev@arrow.apache.org > Subject: Re: user-defined Python-based data-sources in Arrow

Re: user-defined Python-based data-sources in Arrow

2022-06-23 Thread Weston Pace
g > schema-carrying tabular data. > > > Yaron. > > From: Li Jin > Sent: Wednesday, June 22, 2022 2:50 PM > To: dev@arrow.apache.org > Subject: Re: user-defined Python-based data-sources in Arrow > > Yaron, > > Do you mind al

Re: user-defined Python-based data-sources in Arrow

2022-06-22 Thread Yaron Gvili
-source function as producing schema-carrying tabular data. Yaron. From: Li Jin Sent: Wednesday, June 22, 2022 2:50 PM To: dev@arrow.apache.org Subject: Re: user-defined Python-based data-sources in Arrow Yaron, Do you mind also linking the previous mailing list

Re: user-defined Python-based data-sources in Arrow

2022-06-22 Thread Li Jin
Yaron, Do you mind also linking the previous mailing list discussion here? On Wed, Jun 22, 2022 at 11:40 AM Yaron Gvili wrote: > Hi, > > I'd like to get the community's feedback about a design proposal > (discussed below) for integrating user-defined Python-based data-source

user-defined Python-based data-sources in Arrow

2022-06-22 Thread Yaron Gvili
Hi, I'd like to get the community's feedback about a design proposal (discussed below) for integrating user-defined Python-based data-sources in Arrow. This is part of a larger project I'm working on to provide end-to-end (Ibis/Ibis-Substrait/Arrow) support for such data-sources. A user