Le 17/02/2021 à 05:20, Micah Kornfield a écrit :
>>
>> If a method could potentially run some kind of long term blocking I/O
>> wait then yes. So reading / writing tables & datasets, IPC,
>> filesystem APIs, etc. will all need to adapt. It doesn't have to be
>> all at once. CPU only functions
>
> If a method could potentially run some kind of long term blocking I/O
> wait then yes. So reading / writing tables & datasets, IPC,
> filesystem APIs, etc. will all need to adapt. It doesn't have to be
> all at once. CPU only functions would remain as they are. So table
> manipulation, comp
Thanks for this useful writeup and the enseuing discussion. What you're
proposing basically looks sound to me.
Regards
Antoine.
Le 16/02/2021 à 09:29, Weston Pace a écrit :
> Thanks for the input. I appreciate you both taking the time to look
> through this. I'll consolidate the points her
Thanks for the input. I appreciate you both taking the time to look
through this. I'll consolidate the points here.
>From Wes:
> I hypothesize that the bottom of the stack is a thread pool with a
queue-per-thread that implements work stealing.
Yes. I think there may be two pools (one for I/O
I took a pass through this, thank you for a good discussion of the
alternative. One thing that I don't quite understand with this proposal is
the scope? Is the intention that most APIs will eventually work with
Futures instead of raw return values (i.e. returning a Table or Record
batch will neve
hi Weston,
Thanks for putting this comprehensive and informative document together.
There are several layers of problems to consider, just thinking out loud:
* I hypothesize that the bottom of the stack is a thread pool with a
queue-per-thread that implements work stealing. Some code paths might
This is a follow up to a discussion from last September [3]. I've
been investigating Arrow's use of threading and I/O and I believe
there are some improvements that could be made. Arrow is currently
supporting two threading options (single thread and "per-core" thread
pool). Both of these approa
