Paul Weiss created SPARK-10399:
----------------------------------
Summary: Off Heap Memory Access for non-JVM libraries (C++)
Key: SPARK-10399
URL: https://issues.apache.org/jira/browse/SPARK-10399
Project: Spark
Issue Type: Improvement
Components: Spark Core
Reporter: Paul Weiss
*Summary*
Provide direct off-heap memory access to an external non-JVM program such as a
c++ library within the Spark running JVM/executor. As Spark moves to storing
all data into off heap memory it makes sense to provide access points to the
memory for non-JVM programs.
----
*Assumptions*
* Zero copies will be made during the call into non-JVM library
* Access into non-JVM libraries will be accomplished via JNI
* A generic JNI interface will be created so that developers will not need to
deal with the raw JNI call
* C++ will be the initial target non-JVM use case
* memory management will remain on the JVM/Spark side
* the API from C++ will be similar to dataframes as much as feasible and NOT
require expert knowledge of JNI
* Data organization and layout will support complex (multi-type, nested, etc.)
types
----
*Design*
* Initially Spark JVM -> non-JVM will be supported
* Creating an embedded JVM with Spark running from a non-JVM program is
initially out of scope
----
*Technical*
* GetDirectBufferAddress is the JNI call used to access byte buffer without copy
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
