GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/22917
[SPARK-25827][CORE] Encrypted blocks can be over 2GB.
## What changes were proposed in this pull request?
Avoid converting encrypted bocks to regular ByteBuffers.
Also updates some TODOs with links to a SPARK-25905 for improving the
handling here.
## How was this patch tested?
Tested on a cluster with encrypted data > 2GB (after SPARK-25904 was
applied as well).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/squito/spark real_SPARK-25827
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22917.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22917
----
commit 5b432e7184130c0264d330f3068a80f1b3a4bb61
Author: Imran Rashid <irashid@...>
Date: 2018-10-31T18:23:32Z
[SPARK-25855][CORE] Encrypted blocks can be over 2GB.
Avoid converting encrypted bocks to regular ByteBuffers.
Also updates some TODOs with links to a SPARK-25905 for improving the
handling here.
Tested on a cluster with encrypted data > 2GB (after SPARK-25904 was
applied as well).
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
