GitHub user lipzhu opened a pull request: https://github.com/apache/spark/pull/22765
[SPARK-25757][Build] Upgrade netty-all from 4.1.17.Final to 4.1.30.Final ## What changes were proposed in this pull request? Upgrade netty dependency from 4.1.17 to 4.1.30. Explanation: Currently when sending a ChunkedByteBuffer with more than 16 chunks over the network will trigger a "merge" of all the blocks into one big transient array that is then sent over the network. This is problematic as the total memory for all chunks can be high (2GB) and this would then trigger an allocation of 2GB to merge everything, which will create OOM errors. And we can avoid this issue by upgrade the netty. https://github.com/netty/netty/pull/8038 ## How was this patch tested? Manual tests in some spark jobs. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lipzhu/spark SPARK-25757 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22765.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22765 ---- commit 0753c0fe907070b50812b3fb40868dfcdb2bd969 Author: Zhu, Lipeng <lipzhu@...> Date: 2018-10-17T11:58:05Z upgrade netty-all from 4.1.17.Final to 4.1.30.Final ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org