Re: How to access the off-heap representation of cached data in Spark 2.0

2016-05-29 Thread jpivar...@gmail.com
Okay, that makes sense. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/How-to-access-the-off-heap-representation-of-cached-data-in-Spark-2-0-tp17701p17723.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. ---

Re: How to access the off-heap representation of cached data in Spark 2.0

2016-05-29 Thread Jacek Laskowski
On Sun, May 29, 2016 at 5:30 PM, jpivar...@gmail.com wrote: > If I find a way to provide > access by modifying Spark source, can I just submit a pull request, or do I > need to be a recognized Spark developer? If so, is there a process for > becoming one? Start a discussion here (or user@spark)

Re: How to access the off-heap representation of cached data in Spark 2.0

2016-05-29 Thread jpivar...@gmail.com
Thanks Jacek and Kazuaki, I guess got the wrong impression about the C++ API from somewhere--- I think I read it in a JIRA wish list. If the byte array is accessed through sun.misc.Unsafe, that's what I mean by off-heap. I found the Platform class, which provides access to the bytes in Unsafe in a