HeartSaVioR commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-653246229
Let's try to overestimate for memory usage, as it's more critical than estimating disk usage (even for disk usage I think overestimating a bit is safer) and might lead to OOME if end users configure too tight on remaining heap memory. zstd deserves to get at least 7x in the latest experiment, right? IMHO probably safer to apply 10x, and also apply 4x for other compression codecs as well. Let's hear @tgravescs voice on this. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
