ndscoder commented on issue #19216:
URL: https://github.com/apache/datafusion/issues/19216#issuecomment-6425583778
A smaller reproduction of this, still present in DataFusion 55 (through
datafusion-python 55.0.0): 2 million rows in memory, two 8-byte columns (about
32 MB), sorted under a 64 MB greedy pool. One partition sorts them every time;
two partitions fail every time (10 of 10 runs each).
```python
import numpy as np
import pyarrow as pa
from datafusion import RuntimeEnvBuilder, SessionConfig, SessionContext
ROWS = 2_000_000
keys = np.random.default_rng(1).integers(0, 10**9, ROWS)
table = pa.table({"k": keys, "v": np.arange(ROWS, dtype=np.float64)})
for partitions in (1, 2):
step = ROWS // partitions
parts = [table.slice(i * step, step).to_batches(max_chunksize=65536) for
i in range(partitions)]
runtime =
RuntimeEnvBuilder().with_disk_manager_os().with_greedy_memory_pool(64 * 2**20)
context =
SessionContext(SessionConfig().with_target_partitions(partitions), runtime)
context.register_record_batches("t", parts)
try:
rows = sum(b.to_pyarrow().num_rows for b in context.sql("SELECT *
FROM t ORDER BY k").execute_stream())
print(partitions, "partitions: sorted", rows)
except Exception as error:
print(partitions, "partitions:", str(error).splitlines()[0])
```
Output:
```
1 partitions: sorted 2000000
2 partitions: DataFusion error: Not enough memory to continue external sort.
Consider increasing the memory limit config: 'datafusion.runtime.memory_limit',
or decreasing the config: 'datafusion.execution.sort_spill_reservation_bytes'.
```
The rest of the error, from one run (the operator that fails varies between
runs: `ExternalSorter[n]` or `ExternalSorterMerge[n]`):
```
caused by
Resources exhausted: Failed to allocate additional 16.4 MB for
ExternalSorter[1] with 0.0 B already allocated for this reservation - 3.0 MB
remain available for the total memory pool: greedy(used: 61.0 MB, pool_size:
64.0 MB)
```
Other things we saw:
- With a 256 MB pool and eight times the rows, two partitions still fail in
about half the runs.
- On a 40-million-row table read from Parquet, 2, 4, 8 and 16 partitions
fail under a 512 MB pool with both the greedy and the fair spill pool, while
one partition sorts it under the same pool.
- Changing `sort_spill_reservation_bytes` (1 MB to 200 MB) did not help,
except a fair pool with a 1 MB reservation once the pool held about 40% of the
rows.
For us this means a bounded ordered export has to run at one partition,
which roughly doubles its time on fast storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
