sunchao commented on code in PR #7316:
URL: https://github.com/apache/datafusion-comet/pull/7316#discussion_r4633430066


##########
native/core/src/execution/planner.rs:
##########
@@ -1853,19 +1856,21 @@ impl PhysicalPlanner {
                 let (scans, shuffle_scans, child) =
                     self.create_plan(&children[0], inputs, partition_count)?;
 
-                let exprs: Result<Vec<PhysicalSortExpr>, ExecutionError> = sort
+                let exprs: Vec<PhysicalSortExpr> = sort
                     .sort_orders
                     .iter()
                     .map(|expr| self.create_sort_expr(expr, child.schema()))
-                    .collect();
+                    .collect::<Result<_, ExecutionError>>()?;
 
                 let fetch = sort.fetch.map(|num| num as usize);
 
-                let sort_plan = SortExec::new(
-                    LexOrdering::new(exprs?).unwrap(),
+                // The sort evaluates its keys on each batch it receives.
+                let input = self.bound_scalar_expansion(

Review Comment:
   [P2] Avoid exposing ordinary sorts to the shared-buffer reservation bug
   
   Could the ordinary-sort path avoid handing `ExternalSorter` zero-copy slices 
of one large allocation until its shared-buffer accounting is fixed? `SELECT 
id, cume_dist() OVER (ORDER BY id) AS cd FROM range(1048576) SORT BY 
concat(CAST(id AS STRING), '!')` reaches a full-partition `WindowAggExec` 
followed by this sort. Even the one-byte `'!'` installs the splitter at 8192 
rows, producing 128 slices of the same roughly 16 MiB buffers. In the pinned 
DataFusion 55.2.0, `insert_batch` reserves `get_record_batch_memory_size(batch) 
+ batch.get_sliced_size()` for each slice independently. That attempts about 2 
GiB plus 16 MiB of reservations, versus about 32 MiB for the unsplit batch, and 
spills when the pool refuses them. The physical input buffers remain shared.
   
   The underlying bug already exists and is tracked in [DataFusion #25804, 
finding 5](https://github.com/apache/datafusion/issues/25804), with 
[#25820](https://github.com/apache/datafusion/pull/25820) providing the 
upstream fix. The regression here is newly exposing a previously unsplit 
window-to-sort path to it, including tiny scalars with no offset-overflow risk. 
Please avoid that new exposure, compact/account the shared inputs correctly, or 
sequence this path after the dependency fix, with coverage for a wide producer 
and a small scalar sort key.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to