Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/19135 Hi @jerryshao thanks for your reviewing. >So it somehow reflects that CPU core contention is the main issue for memory pre-occupation I have modified the code, now it will not request more memory, now it just reduce the times of calling `reserveUnrollMemoryForThisTask` followed by @cloud-fan comments. And also the method is same as `putIteratorAsValues`. Yeah, its impact will be small with small cores. In the above test results, it doesn't bring any regressions, and also better for many cores. For machine learning, we need cache the source data to OFF_HEAP in order to reduce the gc problem. For the configuration, I think the different application scenarios may be different.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org