kumarUjjawal opened a new issue, #25423:
URL: https://github.com/apache/datafusion/issues/25423
### Describe the bug
Case G of `datafusion/sqllogictest/test_files/aggregate_memory_spill.slt`
runs a partial/final aggregation with four partitions, a batch size of 128, and
a 1 MiB greedy memory limit. The query at line 201 fails intermittently in CI:
```
DataFusion error: Resources exhausted: Additional allocation failed for
FinalHashAggregateStream[1] with top memory consumers (across reservations) as:
FinalHashAggregateStream[1]#23297(can spill: true) consumed 340.6 KB, peak
342.6 KB,
FinalHashAggregateStream[3]#23289(can spill: true) consumed 296.0 KB, peak
296.0 KB,
FinalHashAggregateStream[0]#23295(can spill: true) consumed 192.0 KB, peak
269.0 KB,
FinalHashAggregateStream[2]#23299(can spill: true) consumed 192.0 KB, peak
269.0 KB,
AggregateStream[1]#23298(can spill: false) consumed 48.0 B, peak 48.0 B.
Error: Failed to allocate additional 3.0 KB for FinalHashAggregateStream[1]
with 4.6 KB already allocated for this reservation - 2.5 KB remain available
for the total memory pool: greedy(used: 1021.5 KB, pool_size: 1024.0 KB)
[SQL] SELECT count(*), sum(total)
FROM (
SELECT (v * 7) % 100000 AS k, sum(v) AS total
FROM generate_series(1, 100000) AS t(v)
GROUP BY (v * 7) % 100000
)
at test_files/aggregate_memory_spill.slt:201
```
Known failures, each on a merge-queue run whose PR did not change Rust code:
| Date | Run | Job | Failing allocation |
|---|---|---|---|
| 2026-09-12 | [run
34719697435](https://github.com/apache/datafusion/actions/runs/34719697435) for
#25230 | verify benchmark results (amd64) | `FinalHashAggregateStream[3]`: 3.0
KB with 4.6 KB already allocated |
| 2026-09-17 | [run
35240006882](https://github.com/apache/datafusion/actions/runs/35240006882/job/105266488408)
for #25399 | cargo test (amd64) | `FinalHashAggregateStream[1]`: 3.0 KB with
4.6 KB already allocated |
The 2026-09-17 run passed on its second attempt. The same base commit passed
the job in the queue runs immediately before and after it.
### Cause
After a partition spills, `FinalHashAggregateStream` releases its hash table
and replays the spill files. A streaming merge feeds an
`OrderedFinalAggregateStream` in `Sorted` mode, under new reservations of the
same consumer (`switch_to_ordered_final_stream` in `hash_stream.rs`). That
replay stream can spill only in `PartiallySorted` mode
(`ordered_final_stream.rs`), so a failed resize of its table reservation
returns an error.
In the 2026-09-17 failure, the replaying partition's merge buffers held
about 336 KiB, and the hash tables of the other three partitions held the rest
of the pool. Whether a few KiB are free when the replay table grows depends on
thread scheduling.
This is the same failure class as #25047 (`ordered_aggregate_spill.slt`),
which #25252 resolved by capping `datafusion.runtime.max_spill_merge_fan_in` at
2. Case G has no such cap.
### To Reproduce
The failure depends on scheduling and is rare. The CI runs are linked above.
Locally:
```
cargo test -p datafusion-sqllogictest --test sqllogictests --
aggregate_memory_spill
```
Repeat it with several concurrent copies to add CPU contention.
### Expected behavior
Case G has one outcome under a fixed configuration. It passes on every run
and keeps the 1 MiB limit.
### Additional context
Proposed fix, following #25252: set
`datafusion.runtime.max_spill_merge_fan_in = 2` for Case G and reset it at the
end of the file. Smaller merge buffers leave room for the replay table while
other partitions retain state. I will open a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
