[arrow-datafusion] branch main updated: Return error if spill file does not exist in ExternalSorter (#7479)

[alamb]

This is an automated email from the ASF dual-hosted git repository.

alamb pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion.git


The following commit(s) were added to refs/heads/main by this push:
     new a2caad0cb4 Return error if spill file does not exist in ExternalSorter 
(#7479)
a2caad0cb4 is described below

commit a2caad0cb40e136db365f7bbdff3c3c43e0d2737
Author: Liang-Chi Hsieh <vii...@gmail.com>
AuthorDate: Tue Sep 5 10:55:01 2023 -0700

    Return error if spill file does not exist in ExternalSorter (#7479)
---
 datafusion/core/src/physical_plan/sorts/sort.rs | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/datafusion/core/src/physical_plan/sorts/sort.rs 
b/datafusion/core/src/physical_plan/sorts/sort.rs
index 5d23e72fe7..07bb4cd357 100644
--- a/datafusion/core/src/physical_plan/sorts/sort.rs
+++ b/datafusion/core/src/physical_plan/sorts/sort.rs
@@ -328,6 +328,12 @@ impl ExternalSorter {
             }
 
             for spill in self.spills.drain(..) {
+                if !spill.path().exists() {
+                    return Err(DataFusionError::Internal(format!(
+                        "Spill file {:?} does not exist",
+                        spill.path()
+                    )));
+                }
                 let stream = read_spill_as_stream(spill, self.schema.clone())?;
                 streams.push(stream);
             }