GitHub user squito opened a pull request: https://github.com/apache/spark/pull/22916
[SPARK-25855][CORE] Encrypted blocks can be over 2GB. ## What changes were proposed in this pull request? Avoid converting encrypted bocks to regular ByteBuffers. Also updates some TODOs with links to a SPARK-25905 for improving the handling here. ## How was this patch tested? Tested on a cluster with encrypted data > 2GB (after SPARK-25904 was applied as well). You can merge this pull request into a Git repository by running: $ git pull https://github.com/squito/spark real_SPARK-25855 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22916.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22916 ---- commit 5b432e7184130c0264d330f3068a80f1b3a4bb61 Author: Imran Rashid <irashid@...> Date: 2018-10-31T18:23:32Z [SPARK-25855][CORE] Encrypted blocks can be over 2GB. Avoid converting encrypted bocks to regular ByteBuffers. Also updates some TODOs with links to a SPARK-25905 for improving the handling here. Tested on a cluster with encrypted data > 2GB (after SPARK-25904 was applied as well). ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org