Re: Tungsten off heap memory access for C++ libraries

2.0? Even if the above answers my first question, I'd still like to know if the new Spark API will allow RDDs to be /filled/ from the C++ side, as a data source, rather than a derived dataset. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Tu

Re: Tungsten off heap memory access for C++ libraries

using it for consistency, maybe even interoperability. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Tungsten-off-heap-memory-access-for-C-libraries-tp13898p17387.html Sent from the Apache Spark Developers List mailing list archive at Nabb

Re: Tungsten off heap memory access for C++ libraries

/djinni/tree/master/example/localhost For the long deets, see: https://github.com/dropbox/djinni/pull/140 -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Tungsten-off-heap-memory-access-for-C-libraries-tp13898p14427.html Sent from the Apache Spark

Re: Tungsten off heap memory access for C++ libraries

tring using JNI. It's a very messy problem :) > > Was there indeed a JIRA started to track this issue? Can't find it at the > moment ... > > > > -- > View this message in context: > http://apache-spark-developers-list.1001551.n3.nabble.com/Tungsten-off-heap-memor

Re: Tungsten off heap memory access for C++ libraries

into messing with (standard) Java String <-> std::string using JNI. It's a very messy problem :) Was there indeed a JIRA started to track this issue? Can't find it at the moment ... -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Tungsten-off-heap-

Re: Tungsten off heap memory access for C++ libraries

Please do. Thanks. On Mon, Aug 31, 2015 at 5:00 AM, Paul Weiss wrote: > Sounds good, want me to create a jira and link it to SPARK-9697? Will put > down some ideas to start. > On Aug 31, 2015 4:14 AM, "Reynold Xin" wrote: > >> BTW if you are

Re: Tungsten off heap memory access for C++ libraries

Sounds good, want me to create a jira and link it to SPARK-9697? Will put down some ideas to start. On Aug 31, 2015 4:14 AM, "Reynold Xin" wrote: > BTW if you are interested in this, we could definitely get some help in > terms of prototyping the feasibility, i.e. how we can

Re: Tungsten off heap memory access for C++ libraries

On Sun, Aug 30, 2015 at 5:58 AM, Paul Weiss wrote: > > Also, is this work being done on a branch I could look into further and > try out? > > We don't have a branch yet -- because there is no code nor design for this yet. As I said, it is one of the motivations behind

Re: Tungsten off heap memory access for C++ libraries

BTW if you are interested in this, we could definitely get some help in terms of prototyping the feasibility, i.e. how we can have a native (e.g. C++) API for data access shipped with Spark. There are a lot of questions (e.g. build, portability) that need to be answered. On Mon, Aug 31, 2015 at

Re: Tungsten off heap memory access for C++ libraries

Reynold, That is great to hear. Definitely interested in how 2. is being implemented and how it will be exposed in C++. One important aspect of leveraging the off heap memory is how the data is organized as well as being able to easily access it from the C++ side. For example how would you

Tungsten off heap memory access for C++ libraries

Hi, Would the benefits of project tungsten be available for access by non-JVM programs directly into the off-heap memory? Spark using dataframes w/ the tungsten improvements will definitely help analytics within the JVM world but accessing outside 3rd party c++ libraries is a challenge

Re: Tungsten off heap memory access for C++ libraries

I would also like to see data shared off-heap to a 3rd party C++ library with JNI, I think the complications would be how to memory manage this and make sure the 3rd party libraries also adhere to the access contracts as well. Tim On Sat, Aug 29, 2015 at 12:17 PM, Paul Weiss

Re: Tungsten off heap memory access for C++ libraries

Supporting non-JVM code without memory copying and serialization is actually one of the motivations behind Tungsten. We didn't talk much about it since it is not end-user-facing and it is still too early. There are a few challenges still: 1. Spark cannot run entirely in off-heap mode (by entirely