Apache Arrow data in buffer to RDD/DataFrame/Dataset?

In a few earlier posts [ 1 ] [ 2

Re: How to access the off-heap representation of cached data in Spark 2.0

Okay, that makes sense. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/How-to-access-the-off-heap-representation-of-cached-data-in-Spark-2-0-tp17701p17723.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. ---

Re: How to access the off-heap representation of cached data in Spark 2.0

Thanks Jacek and Kazuaki, I guess got the wrong impression about the C++ API from somewhere--- I think I read it in a JIRA wish list. If the byte array is accessed through sun.misc.Unsafe, that's what I mean by off-heap. I found the Platform class, which provides access to the bytes in Unsafe in a

Re: How to access the off-heap representation of cached data in Spark 2.0

Is this not the place to ask such questions? Where can I get a hint as to how to access the new off-heap cache, or C++ API, if it exists? I'm willing to do my own research, but I have to have a place to start. (In fact, this is the first step in that research.) Thanks, -- Jim -- View this mess

How to access the off-heap representation of cached data in Spark 2.0

Following up on an earlier thread , I would like to access the off-heap representation of cached data in Spark 2.0 in order to see how Spark might be linked to physics softwar

Re: Tungsten off heap memory access for C++ libraries

jpivar...@gmail.com wrote > P.S. Concerning Java/C++ bindings, there are many. I tried JNI, JNA, > BridJ, and JavaCPP personally, but in the end picked JNA because of its > (comparatively) large user base. If Spark will be using Djinni, that could > be a symmetry-breaking considerat

Re: Tungsten off heap memory access for C++ libraries

Hi, I'm coming from the particle physics community and I'm also very interested in the development of this project. We have a huge C++ codebase and would like to start using the higher-level abstractions of Spark in our data analyses. To this end, I've been developing code that copies data from ou