2010YOUY01 commented on code in PR #15700:
URL: https://github.com/apache/datafusion/pull/15700#discussion_r2041372025
##########
datafusion/physical-plan/src/sorts/sort.rs:
##########
@@ -431,12 +422,16 @@ impl ExternalSorter {
let batches_to_spill = std::mem::take(globally_sorted_batches);
self.reservation.free();
- let in_progress_file =
self.in_progress_spill_file.as_mut().ok_or_else(|| {
- internal_datafusion_err!("In-progress spill file should be
initialized")
- })?;
+ let (in_progress_file, max_record_batch_size) =
+ self.in_progress_spill_file.as_mut().ok_or_else(|| {
+ internal_datafusion_err!("In-progress spill file should be
initialized")
+ })?;
for batch in batches_to_spill {
in_progress_file.append_batch(&batch)?;
+
+ *max_record_batch_size =
+ (*max_record_batch_size).max(batch.get_actually_used_size());
Review Comment:
I think it's not realistic to correctly know a batch's size after a
roundtrip of spilling and reading back, with this `get_actually_used_size()`
implementation. The actual implementation might give us some surprise. The
implementation can get even more complex in the future, for example we might
implement extra encodings for
https://github.com/apache/datafusion/issues/14078, and the memory size of a
batch after reading back can be harder to estimate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
