Wes McKinney created ARROW-7001:
-----------------------------------
Summary: [C++] Develop threading APIs to accommodate nested
parallelism
Key: ARROW-7001
URL: https://issues.apache.org/jira/browse/ARROW-7001
Project: Apache Arrow
Issue Type: New Feature
Components: C++
Reporter: Wes McKinney
Tasks invoked in parallel may be able to submit their own subtasks, which in
OpenMP and TBB documentation is often called "nested parallelism".
If a task blocks on the completion of subtasks, then outright deadlocks are
possible -- running tasks are all blocking on their subtasks, but the thread
pool will not schedule any further tasks.
I suggest that such code have a way to indicate to the thread pool (if one is
passed in) that it is blocking on the completion of other tasks so that further
tasks can be run while the task waits for its child tasks to complete. One
possible way to do this is to have a floating "soft limit" for concurrent tasks
that can be incremented when tasks are waiting.
So if we normally allow 8 concurrent tasks, then this can be temporarily
increased for each "suspended" task. Preferably we would provide some way for
the dependent task group to "awaken" the suspended task so that it does not
have to do any work while waiting for the task group to finish
Note this feature can also be used in tasks that are waiting for IO calls
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
