Re: Make off-heap store pluggable

, 2015 at 2:46 AM, Alexey Goncharuk alexey.goncha...@gmail.com wrote: Hello Spark community, I was looking through the code in order to understand better how RDD is persisted to Tachyon off-heap filesystem. It looks like that the Tachyon filesystem is hard-coded and there is no way to switch

Re: Make off-heap store pluggable

2015-07-20 23:29 GMT-07:00 Matei Zaharia matei.zaha...@gmail.com: I agree with this -- basically, to build on Reynold's point, you should be able to get almost the same performance by implementing either the Hadoop FileSystem API or the Spark Data Source API over Ignite in the right way. This

Re: Make off-heap store pluggable

2015-07-20 21:40 GMT-07:00 Reynold Xin r...@databricks.com: I sent it prematurely. They are already pluggable, or at least in the process to be more pluggable. In 1.4, instead of calling the external system's API directly, we added an API for that. There is a patch to add support for HDFS

Make off-heap store pluggable

Hello Spark community, I was looking through the code in order to understand better how RDD is persisted to Tachyon off-heap filesystem. It looks like that the Tachyon filesystem is hard-coded and there is no way to switch to another in-memory filesystem. I think it would be great if the

Sharing memory across applications/integration

Hello Spark community, I am currently trying to implement a proof-of-concept RDD that will allow to integrate Apache Spark and Apache Ignite (incubating) [1]. My original idea was to embed an Ignite node in Spark's worker process, in order for the user code to have a direct access to in-memory