andygrove commented on code in PR #2780:
URL: 
https://github.com/apache/datafusion-ballista/pull/2780#discussion_r4579320991


##########
ballista/executor/src/execution_engine.rs:
##########
@@ -538,7 +541,12 @@ async fn drive_shuffle_writer_stage(
     for n in 0..k {
         let plan = plan.clone();
         let ctx = context.clone();
-        stream_futures.push(tokio::spawn(async move {
+        // Dropping this task stops the writer's input. A writer that streams
+        // to disk on a blocking thread still finishes the file it is writing,
+        // so a cancelled Ballista task can leave a short but valid IPC file

Review Comment:
   Nit: your reply mentioned the range writer can also leave its data file 
without an index file. Maybe add that to this comment so the next reader 
doesn't have to rediscover it?



##########
ballista/core/src/execution_plans/shuffle_writer.rs:
##########
@@ -953,27 +1048,24 @@ impl ExecutionPlan for ShuffleWriterExec {
         partition: usize,
         context: Arc<TaskContext>,
     ) -> Result<SendableRecordBatchStream> {
-        let mut guard = self.state.lock().map_err(|_| {
-            DataFusionError::Internal("ShuffleWriterExec state mutex 
poisoned".to_owned())
-        })?;
-        let state = &mut *guard;
-        let rx = take_handoff(
-            "ShuffleWriterExec",
-            &mut state.initialized,
-            &mut state.handoffs,
-            partition,
-            |senders| {
+        let handoff = self
+            .state
+            .lock()
+            .map_err(|_| {
+                DataFusionError::Internal(
+                    "ShuffleWriterExec state mutex poisoned".to_owned(),
+                )
+            })?
+            .take_handoff("ShuffleWriterExec", partition, |senders| {
                 let writer = self.clone();
-                tokio::spawn(async move {
+                SpawnedTask::spawn(async move {

Review Comment:
   Since this touches the shuffle write path, could you share a quick 
before/after against `main`? Something like `cargo run --release --bin tpch -- 
benchmark ballista --host localhost --port 50050 --iterations 3 --path <tpch 
parquet dir> --format parquet --query <n>` at SF10 for a couple of 
shuffle-heavy queries (for example Q5 and Q9) would be plenty. I don't expect a 
difference, it's just a sanity check.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to