0xdeafbeef commented on code in PR #2780:
URL:
https://github.com/apache/datafusion-ballista/pull/2780#discussion_r4585249205
##########
ballista/executor/src/execution_engine.rs:
##########
@@ -538,7 +541,12 @@ async fn drive_shuffle_writer_stage(
for n in 0..k {
let plan = plan.clone();
let ctx = context.clone();
- stream_futures.push(tokio::spawn(async move {
+ // Dropping this task stops the writer's input. A writer that streams
+ // to disk on a blocking thread still finishes the file it is writing,
+ // so a cancelled Ballista task can leave a short but valid IPC file
Review Comment:
added a comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]