cjolowicz commented on issue #12136: URL: https://github.com/apache/datafusion/issues/12136#issuecomment-2320602379
> I couldn't get the sort to complete because the pod froze when filesystem usage grew to 1TB. I could reproduce the pod freezing up at 1TB disk usage. This happened within a minute of spawning 500 threads. All threads were blocked on a mutex (`futex_wait`) except one in `epoll` with an inode fd. Maybe deadlock at the start of the merge phase? > 1TB seems a bit much for sorting a 100GiB partition. It actually makes sense because spill files use Apache IPC format without compression, while the partition uses Parquet files with Snappy compression. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
