westonpace commented on a change in pull request #9615:
URL: https://github.com/apache/arrow/pull/9615#discussion_r628438007
##########
File path: r/src/r_to_arrow.cpp
##########
@@ -46,6 +50,41 @@ using internal::MakeConverter;
namespace r {
+class RTasks {
+ public:
+ using Task = std::function<Status()>;
+
+ RTasks()
+ : parallel_tasks(arrow::internal::TaskGroup::MakeThreaded(
+ arrow::internal::GetCpuThreadPool())) {}
+
+ Status Finish() {
+ Status status = Status::OK();
+
+ // run the delayed tasks now
+ for (auto& task : delayed_serial_tasks) {
+ status &= task();
+ if (!status.ok()) break;
+ }
+
+ // then wait for the parallel tasks to finish
+ status &= parallel_tasks->Finish();
Review comment:
Scanning through quickly I just thought "it would be nice" but now
thinking about implementing it I think "but it sure would be tricky" :laughing:
I think you would need to use a stop token of some kind. Either
`arrow::StopToken` in `arrow/util/cancel.h` or an atomic bool of some kind that
is set by the serial tasks.
`TaskGroup::MakeThreaded` can take in a stop token as well. That might be
easiest:
- Create a `StopSource`
- Get a `StopToken` from the source
- Pass the `StopToken` into `TaskGroup::MakeThreaded`
- After doing the serial work, if there is an error, call `RequestStop` on
the source.
That would cancel any conversion tasks that are scheduled but not yet
executing. Any tasks currently executing would simply have to run until
completion. If you really wanted to make it responsive then you could pass the
`StopToken` into your conversion functions and check it periodically to see if
a stop was requested and, if so, bail early.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
