coderfender commented on code in PR #1916:
URL:
https://github.com/apache/datafusion-ballista/pull/1916#discussion_r4157131266
##########
ballista/core/src/execution_plans/sort_shuffle/writer.rs:
##########
@@ -630,10 +644,14 @@ impl SortShuffleWriterExec {
// `MemoryPool` as the sole spill trigger.
let memory_limit = config.memory_limit_per_task_bytes;
let per_task_budget_enabled = memory_limit > 0;
+ let mut null_counts = vec![0u64; schema.fields().len()];
while let Some(result) = stream.next().await {
let input_batch = result?;
metrics.input_rows.add(input_batch.num_rows());
+ for (i, col) in input_batch.columns().iter().enumerate() {
+ null_counts[i] += col.null_count() as u64;
Review Comment:
Yes. this is a great catch Andy. I've updated code to rely on
`logical_null_count` instead of `null_count`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
