ivankelly opened a new issue, #18011:
URL: https://github.com/apache/datafusion/issues/18011
### Describe the bug
Doing an external sort is using over 10 times the amount of disk relative to
the raw data.
### To Reproduce
Create a test file in datafusion-cli with
```
copy (select digest(value::string, 'md5') as key, repeat('x', 10000) as text
from generate_series(1000000)) to 'data.parquet';
```
The file itself is fairly small, due to compression, but raw data should be
around 10GB.
Then get the repro code from from https://github.com/ivankelly/datafusion
```
target/release/out_of_disk gen-input --input data.parquet --output
data-1m-1.parquet --sort-mem-limit 1000000000 --sort-partitions 2
--sort-reservation-bytes 1000000
...
Error: Resources exhausted: The used disk space during the spilling process
has exceeded the allowable limit of 100.0 GB. Try increasing the
`max_temp_directory_size` in the disk manager configuration.
```
### Expected behavior
I would expect external sort to use 2-3x the amount of disk as exists in the
raw data. But this is going to 10x and probably more. Are spill files being
left behind after they're no longer needed?
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
