[
https://issues.apache.org/jira/browse/SPARK-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16607532#comment-16607532
]
Apache Spark commented on SPARK-10399:
--------------------------------------
User 'gatorsmile' has created a pull request for this issue:
https://github.com/apache/spark/pull/22361
> Off Heap Memory Access for non-JVM libraries (C++)
> --------------------------------------------------
>
> Key: SPARK-10399
> URL: https://issues.apache.org/jira/browse/SPARK-10399
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: Paul Weiss
> Priority: Major
>
> *Summary*
> Provide direct off-heap memory access to an external non-JVM program such as
> a c++ library within the Spark running JVM/executor. As Spark moves to
> storing all data into off heap memory it makes sense to provide access points
> to the memory for non-JVM programs.
> ----
> *Assumptions*
> * Zero copies will be made during the call into non-JVM library
> * Access into non-JVM libraries will be accomplished via JNI
> * A generic JNI interface will be created so that developers will not need to
> deal with the raw JNI call
> * C++ will be the initial target non-JVM use case
> * memory management will remain on the JVM/Spark side
> * the API from C++ will be similar to dataframes as much as feasible and NOT
> require expert knowledge of JNI
> * Data organization and layout will support complex (multi-type, nested,
> etc.) types
> ----
> *Design*
> * Initially Spark JVM -> non-JVM will be supported
> * Creating an embedded JVM with Spark running from a non-JVM program is
> initially out of scope
> ----
> *Technical*
> * GetDirectBufferAddress is the JNI call used to access byte buffer without
> copy
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
