Re: How to access the off-heap representation of cached data in Spark 2.0

Okay, that makes sense. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/How-to-access-the-off-heap-representation-of-cached-data-in-Spark-2-0-tp17701p17723.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. ---

Re: How to access the off-heap representation of cached data in Spark 2.0

On Sun, May 29, 2016 at 5:30 PM, jpivar...@gmail.com wrote: > If I find a way to provide > access by modifying Spark source, can I just submit a pull request, or do I > need to be a recognized Spark developer? If so, is there a process for > becoming one? Start a discussion here (or user@spark)

Re: How to access the off-heap representation of cached data in Spark 2.0

Thanks Jacek and Kazuaki, I guess got the wrong impression about the C++ API from somewhere--- I think I read it in a JIRA wish list. If the byte array is accessed through sun.misc.Unsafe, that's what I mean by off-heap. I found the Platform class, which provides access to the bytes in Unsafe in a

Re: How to access the off-heap representation of cached data in Spark 2.0

2016/05/29 08:18 Subject: Re: How to access the off-heap representation of cached data in Spark 2.0 Hi Jim, There's no C++ API in Spark to access the off-heap data. Moreover, I also think "off-heap" has an overloaded meaning in Spark - for tungsten and to persist you

Re: How to access the off-heap representation of cached data in Spark 2.0

Hi Jim, There's no C++ API in Spark to access the off-heap data. Moreover, I also think "off-heap" has an overloaded meaning in Spark - for tungsten and to persist your data off-heap (it's all about memory but for different purposes and with client- and internal API). That's my limited understand

Re: How to access the off-heap representation of cached data in Spark 2.0

Is this not the place to ask such questions? Where can I get a hint as to how to access the new off-heap cache, or C++ API, if it exists? I'm willing to do my own research, but I have to have a place to start. (In fact, this is the first step in that research.) Thanks, -- Jim -- View this mess

How to access the off-heap representation of cached data in Spark 2.0

Following up on an earlier thread , I would like to access the off-heap representation of cached data in Spark 2.0 in order to see how Spark might be linked to physics softwar